miRNAselector: Basic Functionality Tutorial.

Konrad Stawiski (konrad@konsta.com.pl), Marcin Kaszkowiak

Department of Biostatistics and Translational Medicine
Medical University of Lodz, Poland
Mazowiecka 15, 92-215 Lodz
tel: +48 42 272 53 85, www: http://biostat.umed.pl

Introduction

Setup script

The package miRNAselector has a lot of requirements that are nessesary to run all the experiments. The script below will allow to install most of them. It is highly recommended to install those packages using the code below.

What should be done? - If you have Nvidia GPU - install CUDA and set gpu=T in the setup script below.

readLines("setup.R") %>% paste0(collapse = "\n") %>% cat
#> install.packages("devtools",repos = "http://cran.r-project.org")
#> install.packages("BiocManager",repos = "http://cran.r-project.org")
#> suppressMessages(library(BiocManager))
#> BiocManager::install(c("reticulate","devtools","plyr","dplyr","edgeR","epiDisplay","rsq","MASS","Biocomb","caret","dplyr",
#>                        "pROC","ggplot2","DMwR", "doParallel", "Boruta", "spFSR", "varSelRF", "stringr", "psych", "C50", "randomForest",
#>                        "foreach","data.table", "ROSE", "deepnet", "gridExtra", "stargazer","gplots","My.stepwise","snow",
#>                        "calibrate", "ggrepel", "networkD3", "VennDiagram","RSNNS", "kernlab", "car", "PairedData",
#>                        "profileR","classInt","kernlab","xgboost", "keras", "tidyverse", "cutpointr","tibble","tidyr", "rpart", "party", "mgcv", "GDCRNATools",
#>                        "imputeMissings", "visdat", "naniar", "stringr", "doSNOW", "R.utils"))
#> remotes::install_github("STATWORX/bounceR", force = T)
#> #remotes::install_github("rstudio/reticulate")
#> remotes::install_github("vqv/ggbiplot", force = T)
#> suppressMessages(library(keras))
#> install_keras()
#> 
#> 
#> remotes::install_github("kstawiski/miRNAselector", force = T) # Install our package.

This code does not cover the installation of mxnet, which can be used for benchmarking of the selected miRNA sets.

Getting the data

In this showcase we will use TCGA-derived data.

miRNAselecter has built-it functions for downloading of miRNA-seq data from TCGA. By default all projects are downloaded and processed using 2 functions as shown below.

ks.download_tissue_miRNA_data_from_TCGA()
ks.process_tissue_miRNA_TCGA(remove_miRNAs_with_null_var = T)

Both of those function produce 2 files: tissue_miRNA_counts.csv and tissue_miRNA_logtpm.csv. First of those files contains metadata and raw counts as declared in TCGA. The second is are log-transformed transcripts-per-million (TPM) counts. Let’s load counts files and see what sample do we have.

suppressWarnings(suppressMessages(library(data.table)))
suppressWarnings(suppressMessages(library(knitr)))
orginal_TCGA_data = fread("tissue_miRNA_counts.csv.gz")
orginal_TCGA_data[orginal_TCGA_data == ""] = NA
kable(table(orginal_TCGA_data$primary_site, orginal_TCGA_data$sample_type))
PrimaryTumor SolidTissueNormal
Adrenal gland 80 0
Bladder 409 19
Brain 512 5
Breast 1078 104
Bronchus and lung 991 91
Cervix uteri 307 3
Corpus uteri|Stomach|Other and unspecified parts of tongue|Meninges|Other and unspecified male genital organs|Colon|Connective, subcutaneous and other soft tissues|Bones, joints and articular cartilage of limbs|Ovary|Retroperitoneum and peritoneum|Peripheral nerves and autonomic nervous system|Uterus, NOS|Kidney 259 0
Corpus uteri|Uterus, NOS 538 33
Esophagus|Stomach 184 13
Eye and adnexa 80 0
Heart, mediastinum, and pleura|Bronchus and lung 87 0
Heart, mediastinum, and pleura|Other endocrine glands and related structures|Adrenal gland|Connective, subcutaneous and other soft tissues|Other and ill-defined sites|Spinal cord, cranial nerves, and other parts of central nervous system|Retroperitoneum and peritoneum 179 3
Heart, mediastinum, and pleura|Testis|Stomach|Lymph nodes|Bones, joints and articular cartilage of other and unspecified sites|Brain|Thyroid gland|Small intestine|Colon|Connective, subcutaneous and other soft tissues|Other and unspecified major salivary glands|Retroperitoneum and peritoneum|Hematopoietic and reticuloendothelial systems|Breast 47 0
Heart, mediastinum, and pleura|Thymus 124 2
Kidney 873 130
Liver and intrahepatic bile ducts 372 50
Other and ill-defined sites in lip, oral cavity and pharynx|Palate|Other and unspecified parts of tongue|Hypopharynx|Tonsil|Oropharynx|Larynx|Other and unspecified parts of mouth|Gum|Floor of mouth|Bones, joints and articular cartilage of other and unspecified sites|Lip|Base of tongue 523 44
Other and unspecified parts of biliary tract|Gallbladder|Liver and intrahepatic bile ducts 36 9
Ovary 489 0
Pancreas 178 4
Prostate gland 494 52
Rectosigmoid junction|Colon 444 8
Rectosigmoid junction|Unknown|Rectum|Colon|Connective, subcutaneous and other soft tissues 161 3
Skin 97 2
Stomach 436 41
Testis 150 0
Thyroid gland 506 59
Uterus, NOS 57 0

Let’s consider a following exemplary problem..

We want to find the set of miRNAs the most specific to pancreatic cancer. We see that there are 178 cases of pancreatic cancer miRNA-seq results and only 4 solid tissue normal cases. However, we have multiple other normal miRNA-seq results that could be incorporated in the analysis. Let’s filter and label the samples of interest.

suppressWarnings(suppressMessages(library(dplyr)))

cancer_cases = filter(orginal_TCGA_data, primary_site == "Pancreas" & sample_type == "PrimaryTumor")
control_cases = filter(orginal_TCGA_data, sample_type == "SolidTissueNormal")

The pipeline requires the variable Class to be present in the dataset. This variable has to be present and have only 2 levels: Cancer and Control.

cancer_cases$Class = "Cancer"
control_cases$Class = "Control"

dataset = rbind(cancer_cases, control_cases)

kable(table(dataset$Class), col.names = c("Class", "Number of cases"))
Class Number of cases
Cancer 178
Control 675

Let’s explore some of the associations between the group.

boxplot(dataset$age_at_diagnosis ~ dataset$Class)

t.test(dataset$age_at_diagnosis ~ dataset$Class)
#> 
#>  Welch Two Sample t-test
#> 
#> data:  dataset$age_at_diagnosis by dataset$Class
#> t = 3.88, df = 360.55, p-value = 0.0001242
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#>   693.0905 2117.7294
#> sample estimates:
#>  mean in group Cancer mean in group Control 
#>              23782.23              22376.82
kable(table(dataset$gender.x, dataset$Class))
Cancer Control
female 80 334
male 98 325
chisq.test(dataset$gender.x, dataset$Class)
#> 
#>  Pearson's Chi-squared test with Yates' continuity correction
#> 
#> data:  dataset$gender.x and dataset$Class
#> X-squared = 1.6241, df = 1, p-value = 0.2025

There is the stistically significant difference in age between classess. The gender was not associated with class. Let’s do propensity score matching to balance the sets.

old_dataset = dataset  # backup
dataset = dataset[grepl("Adenocarcinomas", dataset$disease_type), ]
match_by = c("age_at_diagnosis", "gender.x")
tempdane = dplyr::select(dataset, match_by)
tempdane$Class = ifelse(dataset$Class == "Cancer", TRUE, FALSE)
suppressMessages(library(mice))
suppressMessages(library(MatchIt))
temp1 = mice(tempdane, m = 1)
#> 
#>  iter imp variable
#>   1   1  age_at_diagnosis
#>   2   1  age_at_diagnosis
#>   3   1  age_at_diagnosis
#>   4   1  age_at_diagnosis
#>   5   1  age_at_diagnosis
temp2 = temp1$data
temp3 = mice::complete(temp1)
temp3 = temp3[complete.cases(temp3), ]
tempform = ks.create_miRNA_formula(match_by)
mod_match <- matchit(tempform, data = temp3)
newdata = match.data(mod_match)
dataset = dataset[as.numeric(rownames(newdata)), ]

boxplot(dataset$age_at_diagnosis ~ dataset$Class)

t.test(dataset$age_at_diagnosis ~ dataset$Class)
#> 
#>  Welch Two Sample t-test
#> 
#> data:  dataset$age_at_diagnosis by dataset$Class
#> t = 0.090876, df = 351.63, p-value = 0.9276
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#>  -827.5374  907.7172
#> sample estimates:
#>  mean in group Cancer mean in group Control 
#>              23782.23              23742.14
kable(table(dataset$gender.x, dataset$Class))
Cancer Control
female 80 79
male 98 99
chisq.test(dataset$gender.x, dataset$Class)
#> 
#>  Pearson's Chi-squared test with Yates' continuity correction
#> 
#> data:  dataset$gender.x and dataset$Class
#> X-squared = 0, df = 1, p-value = 1
fwrite(dataset, "balanced_dataset.csv.gz")

Dataset is a bit better balanced now. In the next steps we will: 1. In order to stay consistent between different datasets we will use ks.correct_miRNA_names() to unify the miRNA names between different versions of miRbase. 2. We will perform standard filtering, log-transormation and TPM-normalization.

dataset = ks.correct_miRNA_names(dataset)
danex = dplyr::select(dataset, starts_with("hsa"))  # Create data.frame or matrix with miRNA counts with miRNAs in columns and cases in rows.
metadane = dplyr::select(dataset, -starts_with("hsa"))  # Metadata with 'Class' variables.
kable(table(metadane$Class))  # Let's be sure that 'Class' variable is correct and contains only 'Cancer' and 'Control' cases.
Var1 Freq
Cancer 178
Control 178
ttpm = ks.counts_to_log10tpm(danex, metadane, ids = metadane$sample, filtr = T, filtr_minimalcounts = 100, filtr_howmany = 1/3)  # We will leave only the miRNAs which apeared with at least 100 counts in 1/3 of cases.
#> 
#> DGEList unfiltered object with TPM was saved as TPM_DGEList.rds.
#> DGEList filtered object with TPM was saved as TPM_DGEList_filtered.rds.
#> (After filtering) miRNAs left: 166 | filtered out: 2418.
#> Returned data are log10(TPM).

After application of filter there are still 166 miRNAs left. The filter was applied to ensure our potential diagnostic test may not relay strictly on miRNA-seq data, but the creation of much cheaper qPCR test will be possible.

In the next step we will devide the dataset into training, testing and validation datasets. We strongly belive that hold-out validation is the most redundant validation method and although miRNAselector supports cross-validation, the hold-out validation is set by default in most cases. Thus, the rest of the analysis is dependent of existance of 3 seperate datasets:

The best signiture (best set of miRNAs for diagnostic test) can be selected based on all 3 datasets, 2 datasets or only validation set. The process of best signiture selection will be discussed below.

The split can be prepared manually by user (the pipeline expects to find mixed_*.csv files in working directory) or in a convinient way using ks.prepare_split(). Let’s do it now.

mixed = ks.prepare_split(metadane = metadane, ttpm = ttpm, train_proc = 0.6)
mixed = fread("mixed.csv")
kable(table(mixed$Class, mixed$mix))
test train valid
Cancer 36 107 35
Control 36 107 35
kable(cbind(mixed[1:10, c(100:105)], Class = mixed[1:10, "Class"]))
hsa.miR.30c.5p hsa.miR.30c.2.3p hsa.miR.30d.5p hsa.miR.139.5p hsa.miR.10a.5p hsa.miR.10b.5p Class.Class
2.598949 1.514444 3.802908 1.555401 4.735516 4.298381 Cancer
2.391663 1.416540 3.656058 1.326116 4.615369 3.946954 Cancer
2.576255 1.566166 3.721311 1.220338 4.385001 3.998914 Cancer
2.260492 1.401010 3.539243 1.924365 3.945605 4.486228 Cancer
2.595315 1.562735 3.571441 1.721510 4.519249 4.126032 Cancer
2.632860 1.723000 3.708020 2.120359 4.875796 3.887056 Cancer
2.747742 2.016887 3.852090 1.533525 3.961177 3.642831 Cancer
3.037578 1.880192 3.960987 1.798007 4.675070 4.133894 Cancer
2.598933 1.291485 3.581714 1.413095 4.580566 3.960672 Cancer
2.829601 1.494520 3.589292 1.755581 4.687104 4.209582 Cancer

We can see that the dataset was devided in balanced way. Now we are ready to move to the analysis…

Basic exploratory analysis

In biomarker studies we relay on validation. We perform hold-out validation, but the signature has to be selected based on trainin dataset only. Including testing and validation dataset in the exploratory analysis could lead to bias. In the following section we show how to use our package to perform quick exploratory analysis of miRNA-seq data.

dane = ks.load_datamix(use_smote_not_rose = T)  # load mixed_*.csv files
train = dane[[1]]
test = dane[[2]]
valid = dane[[3]]
train_smoted = dane[[4]]
trainx = dane[[5]]
trainx_smoted = dane[[6]]  # get the objects from list to make the code more readable.

ks_load_datamix() function loads the data created in preparation phase. It requires the output constructed by ks.prepare_split() function to be placed in working directory (‘wd’), thus files ‘mixed_train.csv’, ‘mixed_test.csv’ and ‘mixed_valid.csv’ have to exist in the directory. For imbalanced data, the fuction can perform balancing using:

  1. ROSE (default): https://journal.r-project.org/archive/2014/RJ-2014-008/RJ-2014-008.pdf - by default we generate 10 * number of cases in orginal dataset.
  2. SMOTE: https://arxiv.org/abs/1106.1813 - by defult we use ‘perc.under=100’ and ‘k=10’.

At the beging of the analysis we usually perform principal component analysis (PCA) to assess for any batch effect, possible outliers and get a general understanding of miRNA profile. This package can construct 2-dimentional biplot and 3-dimentional scatterplot based on the computed components to handle this issue.

pca = ks.PCA(trainx, train$Class)
pca

pca3d = ks.PCA_3D(trainx, train$Class)
pca3d

In the next step we can correct the batch effect for example using ks.combat(). The correction of batch effect is out of the scope of this tutorial.

Usually, the next step in the exploratory analysis is to perform the differential expression analysis. Differential expression in our package focuces of biomarker discovery thus uses t-test with the correction for multiple comparisons. The following table shows signifiant miRNAs after BH correction.

de = ks.miRNA_differential_expression(trainx, train$Class)
sig_de = de %>% dplyr::filter(`p-value BH` <= 0.05) %>% dplyr::arrange(`p-value BH`)  # leave only significant after Benjamini-Hochberg procedure and sort by ascending p-value
ks.table(sig_de)
miR mean logtpm median logtpm SD logtpm cancer mean cancer median cancer SD control mean control median control SD log10FC (subtr estim) log10FC log2FC reversed_log10FC reverse_log2FC p-value p-value Bonferroni p-value Holm -log10(p-value Bonferroni) p-value BH
hsa.miR.21.5p 5.081371 5.120977 0.5409456 5.473043 5.559281 0.3042087 4.689698 4.744446 0.4309906 -0.7833450 0.7833450 2.6022157 0.7833450 2.6022157 0.0000000 0.0000000 0.0000000 32.2312221 0.0000000
hsa.miR.30e.3p 3.654915 3.620465 0.2922656 3.445100 3.469065 0.1835853 3.864731 3.891680 0.2215066 0.4196304 -0.4196304 -1.3939819 -0.4196304 -1.3939819 0.0000000 0.0000000 0.0000000 32.1422635 0.0000000
hsa.miR.30a.3p 3.861758 3.760500 0.5107622 3.494206 3.528067 0.2957390 4.229311 4.315264 0.4049904 0.7351047 -0.7351047 -2.4419651 -0.7351047 -2.4419651 0.0000000 0.0000000 0.0000000 31.8224321 0.0000000
hsa.miR.30c.2.3p 1.791478 1.685323 0.4609305 1.478041 1.473434 0.2548946 2.104915 2.062559 0.4044845 0.6268741 -0.6268741 -2.0824306 -0.6268741 -2.0824306 0.0000000 0.0000000 0.0000000 26.3237118 0.0000000
hsa.miR.132.3p 2.062941 2.061854 0.3206914 2.268277 2.222890 0.2453178 1.857606 1.842071 0.2476985 -0.4106709 0.4106709 1.3642193 0.4106709 1.3642193 0.0000000 0.0000000 0.0000000 23.2892084 0.0000000
hsa.miR.375.3p 3.851381 4.088129 1.1312626 4.572062 4.593208 0.6167635 3.130699 3.420583 1.0689350 -1.4413633 1.4413633 4.7881053 1.4413633 4.7881053 0.0000000 0.0000000 0.0000000 21.6930447 0.0000000
hsa.miR.217.5p 2.065790 1.932856 1.2184284 2.817892 2.596188 1.0079357 1.313689 1.247738 0.9084168 -1.5042026 1.5042026 4.9968529 1.5042026 4.9968529 0.0000000 0.0000000 0.0000000 21.0021448 0.0000000
hsa.miR.139.5p 1.916157 1.919751 0.4638487 1.634747 1.558070 0.3573670 2.197567 2.204857 0.3804627 0.5628209 -0.5628209 -1.8696506 -0.5628209 -1.8696506 0.0000000 0.0000000 0.0000000 20.0671420 0.0000000
hsa.miR.199a.5p 3.034892 3.117770 0.3929526 3.267832 3.318017 0.2924584 2.801951 2.808411 0.3394148 -0.4658802 0.4658802 1.5476204 0.4658802 1.5476204 0.0000000 0.0000000 0.0000000 18.7831163 0.0000000
hsa.miR.338.3p 2.743796 2.778247 0.4190933 2.991348 2.994550 0.3219668 2.496243 2.539407 0.3543332 -0.4951049 0.4951049 1.6447027 0.4951049 1.6447027 0.0000000 0.0000000 0.0000000 18.6530266 0.0000000
hsa.miR.199b.5p 2.146095 2.358829 0.6145137 2.504540 2.573683 0.3567535 1.787650 1.799962 0.6100440 -0.7168901 0.7168901 2.3814574 0.7168901 2.3814574 0.0000000 0.0000000 0.0000000 17.2507100 0.0000000
hsa.miR.127.5p 2.214220 2.327709 0.4803146 2.486170 2.478848 0.3020641 1.942270 1.939357 0.4722596 -0.5438996 0.5438996 1.8067954 0.5438996 1.8067954 0.0000000 0.0000000 0.0000000 16.1634770 0.0000000
hsa.miR.381.3p 1.720930 1.791253 0.4579339 1.974984 1.949285 0.3353016 1.466877 1.476716 0.4226746 -0.5081072 0.5081072 1.6878957 0.5081072 1.6878957 0.0000000 0.0000000 0.0000000 15.6751972 0.0000000
hsa.miR.127.3p 2.789656 2.829601 0.3874414 2.995898 3.006282 0.3147991 2.583413 2.568116 0.3415622 -0.4124850 0.4124850 1.3702456 0.4124850 1.3702456 0.0000000 0.0000000 0.0000000 14.1819049 0.0000000
hsa.miR.199a.3p 3.431280 3.533249 0.3595668 3.620030 3.670480 0.2685999 3.242530 3.285480 0.3401837 -0.3775004 0.3775004 1.2540293 0.3775004 1.2540293 0.0000000 0.0000000 0.0000000 13.5707699 0.0000000
hsa.miR.181b.5p 2.447025 2.523360 0.4131408 2.664941 2.705753 0.2848923 2.229109 2.205351 0.4073655 -0.4358316 0.4358316 1.4478013 0.4358316 1.4478013 0.0000000 0.0000000 0.0000000 13.5990480 0.0000000
hsa.miR.199b.3p 3.430095 3.531945 0.3598114 3.619075 3.669630 0.2686093 3.241115 3.284615 0.3404382 -0.3779599 0.3779599 1.2555557 0.3779599 1.2555557 0.0000000 0.0000000 0.0000000 13.5889555 0.0000000
hsa.miR.194.5p 3.119571 3.359054 0.8329947 3.562675 3.633752 0.4167141 2.676468 2.410622 0.9078933 -0.8862067 0.8862067 2.9439150 0.8862067 2.9439150 0.0000000 0.0000000 0.0000000 13.2390851 0.0000000
hsa.miR.134.5p 2.366486 2.433393 0.4088244 2.572458 2.557931 0.2993653 2.160514 2.115254 0.4007277 -0.4119434 0.4119434 1.3684465 0.4119434 1.3684465 0.0000000 0.0000000 0.0000000 12.1552043 0.0000000
hsa.miR.30a.5p 4.429837 4.356060 0.4519742 4.199892 4.226049 0.2683567 4.659782 4.685628 0.4813796 0.4598896 -0.4598896 -1.5277200 -0.4598896 -1.5277200 0.0000000 0.0000000 0.0000000 12.1052493 0.0000000
hsa.miR.200a.5p 2.480123 2.527551 0.5262405 2.744628 2.777438 0.3622178 2.215619 2.330898 0.5329388 -0.5290086 0.5290086 1.7573286 0.5290086 1.7573286 0.0000000 0.0000000 0.0000000 11.9837197 0.0000000
hsa.miR.136.5p 1.426048 1.539528 0.5852494 1.722033 1.713813 0.2950712 1.130063 1.259715 0.6514071 -0.5919692 0.5919692 1.9664791 0.5919692 1.9664791 0.0000000 0.0000000 0.0000000 11.6576372 0.0000000
hsa.miR.192.5p 3.474000 3.723466 0.8856187 3.920081 3.942126 0.4108526 3.027920 2.831424 1.0027511 -0.8921615 0.8921615 2.9636965 0.8921615 2.9636965 0.0000000 0.0000000 0.0000000 11.4242384 0.0000000
hsa.miR.451a 2.540774 2.525599 0.6339915 2.233131 2.192706 0.4710976 2.848417 2.906464 0.6282318 0.6152860 -0.6152860 -2.0439357 -0.6152860 -2.0439357 0.0000000 0.0000000 0.0000000 11.0384525 0.0000000
hsa.miR.200a.3p 2.330223 2.419049 0.6863313 2.660594 2.674894 0.3970887 1.999851 2.171325 0.7539997 -0.6607432 0.6607432 2.1949415 0.6607432 2.1949415 0.0000000 0.0000000 0.0000000 10.4637303 0.0000000
hsa.miR.141.3p 2.470222 2.587047 0.6758787 2.787653 2.835787 0.4142754 2.152790 2.366831 0.7368061 -0.6348621 0.6348621 2.1089664 0.6348621 2.1089664 0.0000000 0.0000000 0.0000000 9.8966505 0.0000000
hsa.miR.181c.3p 1.549848 1.598609 0.3058360 1.691125 1.690791 0.2009568 1.408572 1.397065 0.3275293 -0.2825529 0.2825529 0.9386203 0.2825529 0.9386203 0.0000000 0.0000000 0.0000000 9.5661290 0.0000000
hsa.miR.30d.5p 3.824938 3.786939 0.2585958 3.705810 3.708020 0.1763818 3.944065 3.950384 0.2731545 0.2382549 -0.2382549 -0.7914657 -0.2382549 -0.7914657 0.0000000 0.0000000 0.0000000 9.5425303 0.0000000
hsa.miR.200b.3p 2.716967 2.769036 0.5121158 2.949316 3.014389 0.3627291 2.484618 2.625241 0.5351976 -0.4646977 0.4646977 1.5436922 0.4646977 1.5436922 0.0000000 0.0000000 0.0000000 9.2128238 0.0000000
hsa.miR.708.3p 1.553047 1.799718 0.7046857 1.878493 1.937453 0.3169286 1.227601 1.186108 0.8267867 -0.6508920 0.6508920 2.1622164 0.6508920 2.1622164 0.0000000 0.0000000 0.0000000 9.1585590 0.0000000
hsa.miR.210.3p 2.239766 2.271507 0.6889295 2.547543 2.542301 0.5385826 1.931988 2.106523 0.6873228 -0.6155552 0.6155552 2.0448301 0.6155552 2.0448301 0.0000000 0.0000000 0.0000000 8.9396288 0.0000000
hsa.miR.486.5p 2.102968 2.062283 0.6191482 1.830616 1.754483 0.4121882 2.375321 2.342782 0.6713086 0.5447046 -0.5447046 -1.8094696 -0.5447046 -1.8094696 0.0000000 0.0000000 0.0000000 8.4375052 0.0000000
hsa.miR.29a.3p 3.841416 3.818064 0.2415932 3.736185 3.745283 0.1796643 3.946646 3.954986 0.2503002 0.2104605 -0.2104605 -0.6991346 -0.2104605 -0.6991346 0.0000000 0.0000000 0.0000000 8.3240291 0.0000000
hsa.miR.378a.3p 2.550650 2.512377 0.3937521 2.378397 2.381114 0.2557418 2.722903 2.778885 0.4315532 0.3445063 -0.3445063 -1.1444252 -0.3445063 -1.1444252 0.0000000 0.0000000 0.0000000 8.2934009 0.0000000
hsa.miR.379.5p 2.828974 2.918613 0.4091041 3.006836 3.034753 0.2966347 2.651112 2.644205 0.4294792 -0.3557240 0.3557240 1.1816895 0.3557240 1.1816895 0.0000000 0.0000000 0.0000000 8.2616630 0.0000000
hsa.miR.141.5p 2.216028 2.392275 0.6525748 2.500775 2.556782 0.3976527 1.931282 2.148570 0.7306892 -0.5694928 0.5694928 1.8918140 0.5694928 1.8918140 0.0000000 0.0000000 0.0000000 8.1778863 0.0000000
hsa.miR.21.3p 2.950511 3.002843 0.4495447 3.143884 3.238660 0.3956108 2.757137 2.698429 0.4172376 -0.3867473 0.3867473 1.2847466 0.3867473 1.2847466 0.0000000 0.0000000 0.0000000 8.1503082 0.0000000
hsa.miR.337.3p 1.498952 1.609902 0.4286748 1.685506 1.704399 0.2599192 1.312398 1.278140 0.4810808 -0.3731072 0.3731072 1.2394354 0.3731072 1.2394354 0.0000000 0.0000000 0.0000000 8.1171714 0.0000000
hsa.miR.144.5p 1.799486 1.840358 0.7365132 1.481479 1.461856 0.4729441 2.117492 2.202314 0.8137450 0.6360122 -0.6360122 -2.1127868 -0.6360122 -2.1127868 0.0000000 0.0000000 0.0000000 8.0043727 0.0000000
hsa.miR.429 1.847904 1.950131 0.7268633 2.162646 2.205144 0.3857231 1.533162 1.702486 0.8443154 -0.6294838 0.6294838 2.0910999 0.6294838 2.0910999 0.0000000 0.0000000 0.0000000 7.8988604 0.0000000
hsa.miR.335.3p 1.773607 1.812886 0.3718545 1.932716 1.892784 0.2426583 1.614497 1.631930 0.4097115 -0.3182195 0.3182195 1.0571023 0.3182195 1.0571023 0.0000000 0.0000000 0.0000000 7.8314098 0.0000000
hsa.miR.126.5p 2.232491 2.222927 0.4152358 2.056938 2.063790 0.2104761 2.408045 2.529680 0.4898455 0.3511067 -0.3511067 -1.1663511 -0.3511067 -1.1663511 0.0000000 0.0000000 0.0000000 7.3936205 0.0000000
hsa.miR.30c.5p 2.808698 2.749100 0.3490929 2.661967 2.661797 0.1963921 2.955429 2.915562 0.4035411 0.2934616 -0.2934616 -0.9748585 -0.2934616 -0.9748585 0.0000000 0.0000000 0.0000000 7.3565035 0.0000000
hsa.miR.181a.3p 2.106380 2.174268 0.3784895 2.261578 2.305816 0.2347854 1.951183 2.028792 0.4290792 -0.3103947 0.3103947 1.0311087 0.3103947 1.0311087 0.0000000 0.0000001 0.0000001 6.9641475 0.0000000
hsa.let.7c.5p 3.419322 3.422382 0.3296842 3.287237 3.332260 0.2860501 3.551407 3.535984 0.3183740 0.2641699 -0.2641699 -0.8775535 -0.2641699 -0.8775535 0.0000000 0.0000002 0.0000001 6.7432319 0.0000000
hsa.miR.181a.5p 3.155295 3.246986 0.4050583 3.316282 3.343112 0.2324404 2.994307 2.991241 0.4725895 -0.3219750 0.3219750 1.0695778 0.3219750 1.0695778 0.0000000 0.0000004 0.0000003 6.3623836 0.0000000
hsa.miR.221.3p 2.290528 2.336670 0.3527519 2.427381 2.408816 0.2861108 2.153674 2.182603 0.3610694 -0.2737072 0.2737072 0.9092357 0.2737072 0.9092357 0.0000000 0.0000007 0.0000005 6.1577028 0.0000000
hsa.miR.93.5p 3.298589 3.328666 0.2589551 3.398683 3.400925 0.1891085 3.198495 3.236571 0.2806412 -0.2001879 0.2001879 0.6650098 0.2001879 0.6650098 0.0000000 0.0000009 0.0000006 6.0433338 0.0000000
hsa.miR.378a.5p 1.765728 1.699123 0.3998851 1.610886 1.612851 0.2430057 1.920570 1.955764 0.4624594 0.3096836 -0.3096836 -1.0287466 -0.3096836 -1.0287466 0.0000000 0.0000011 0.0000008 5.9662895 0.0000000
hsa.miR.146b.5p 2.601785 2.674141 0.4626849 2.779955 2.835954 0.2919913 2.423615 2.450139 0.5299317 -0.3563393 0.3563393 1.1837335 0.3563393 1.1837335 0.0000000 0.0000013 0.0000009 5.8990220 0.0000000
hsa.miR.23a.3p 3.411765 3.464466 0.2170216 3.494149 3.504609 0.1702973 3.329380 3.342163 0.2278979 -0.1647689 0.1647689 0.5473504 0.1647689 0.5473504 0.0000000 0.0000016 0.0000011 5.7880964 0.0000000
hsa.miR.30b.5p 2.622413 2.622930 0.3212403 2.499914 2.507222 0.2120817 2.744913 2.792110 0.3634422 0.2449988 -0.2449988 -0.8138683 -0.2449988 -0.8138683 0.0000000 0.0000017 0.0000012 5.7695073 0.0000000
hsa.miR.625.3p 2.068354 2.104397 0.3502308 2.200772 2.210917 0.2188272 1.935936 1.910435 0.4039743 -0.2648357 0.2648357 0.8797652 0.2648357 0.8797652 0.0000000 0.0000025 0.0000017 5.6075234 0.0000000
hsa.miR.182.5p 3.674023 3.755735 0.4922991 3.858145 3.842168 0.3744427 3.489902 3.616382 0.5275933 -0.3682429 0.3682429 1.2232765 0.3682429 1.2232765 0.0000000 0.0000029 0.0000020 5.5415728 0.0000001
hsa.miR.27a.3p 3.005903 3.044024 0.2647920 3.104509 3.109418 0.2287859 2.907296 2.903005 0.2625226 -0.1972135 0.1972135 0.6551290 0.1972135 0.6551290 0.0000000 0.0000030 0.0000020 5.5228315 0.0000001
hsa.miR.183.5p 3.203446 3.257163 0.4833569 3.381328 3.357706 0.3966813 3.025563 3.143805 0.4982314 -0.3557648 0.3557648 1.1818251 0.3557648 1.1818251 0.0000000 0.0000047 0.0000031 5.3289624 0.0000001
hsa.miR.181c.5p 1.665600 1.682222 0.3601516 1.795267 1.778815 0.2462991 1.535932 1.501756 0.4074728 -0.2593354 0.2593354 0.8614935 0.2593354 0.8614935 0.0000001 0.0000115 0.0000076 4.9383329 0.0000002
hsa.miR.126.3p 3.305734 3.272928 0.4223713 3.157448 3.162873 0.2190488 3.454020 3.630912 0.5158520 0.2965719 -0.2965719 -0.9851906 -0.2965719 -0.9851906 0.0000002 0.0000319 0.0000209 4.4963337 0.0000005
hsa.miR.582.3p 2.087755 2.119436 0.3376004 1.973266 1.974597 0.2964765 2.202245 2.291615 0.3386179 0.2289783 -0.2289783 -0.7606496 -0.2289783 -0.7606496 0.0000004 0.0000584 0.0000380 4.2335770 0.0000010
hsa.miR.143.3p 4.940749 4.860769 0.4133211 4.799815 4.796805 0.2682101 5.081683 4.987285 0.4808783 0.2818686 -0.2818686 -0.9363472 -0.2818686 -0.9363472 0.0000004 0.0000618 0.0000398 4.2090891 0.0000010
hsa.miR.185.5p 1.698820 1.752167 0.3122312 1.803017 1.789962 0.1825995 1.594623 1.585711 0.3750133 -0.2083937 0.2083937 0.6922687 0.2083937 0.6922687 0.0000007 0.0001204 0.0000769 3.9192525 0.0000020
hsa.miR.29c.3p 3.321214 3.327217 0.3591607 3.202955 3.215627 0.2277490 3.439473 3.491354 0.4232090 0.2365180 -0.2365180 -0.7856959 -0.2365180 -0.7856959 0.0000010 0.0001621 0.0001026 3.7900871 0.0000026
hsa.miR.223.3p 2.257390 2.288581 0.4632869 2.408097 2.462940 0.4003810 2.106684 2.111760 0.4744849 -0.3014129 0.3014129 1.0012719 0.3014129 1.0012719 0.0000011 0.0001834 0.0001149 3.7365180 0.0000029
hsa.miR.423.5p 1.545436 1.538957 0.3144486 1.443133 1.425446 0.2069216 1.647738 1.666376 0.3670736 0.2046048 -0.2046048 -0.6796823 -0.2046048 -0.6796823 0.0000013 0.0002154 0.0001337 3.6667565 0.0000034
hsa.miR.151a.5p 1.778564 1.824896 0.2355297 1.854691 1.836098 0.1493639 1.702436 1.739749 0.2783202 -0.1522549 0.1522549 0.5057800 0.1522549 0.5057800 0.0000016 0.0002604 0.0001600 3.5843739 0.0000040
hsa.let.7i.5p 2.599921 2.632392 0.2997045 2.696086 2.717300 0.1738861 2.503755 2.466912 0.3627486 -0.1923305 0.1923305 0.6389080 0.1923305 0.6389080 0.0000020 0.0003296 0.0002005 3.4820413 0.0000050
hsa.miR.26b.5p 2.777405 2.755033 0.2942712 2.683523 2.667076 0.2076405 2.871288 2.871957 0.3363026 0.1877642 -0.1877642 -0.6237392 -0.1877642 -0.6237392 0.0000020 0.0003363 0.0002026 3.4732893 0.0000050
hsa.miR.222.3p 1.679643 1.690400 0.3421027 1.788188 1.797616 0.2933599 1.571098 1.634996 0.3540146 -0.2170897 0.2170897 0.7211563 0.2170897 0.7211563 0.0000021 0.0003468 0.0002068 3.4599760 0.0000051
hsa.miR.320a.3p 2.517259 2.510587 0.2335402 2.443505 2.440882 0.2043264 2.591012 2.575436 0.2384649 0.1475073 -0.1475073 -0.4900086 -0.1475073 -0.4900086 0.0000023 0.0003863 0.0002281 3.4130330 0.0000056
hsa.miR.30e.5p 3.679150 3.640665 0.3123538 3.579560 3.581958 0.1513627 3.778740 3.843580 0.3913006 0.1991799 -0.1991799 -0.6616613 -0.1991799 -0.6616613 0.0000025 0.0004221 0.0002467 3.3745491 0.0000060
hsa.miR.200c.3p 3.479500 3.647500 0.6328315 3.679499 3.713871 0.3714689 3.279502 3.524250 0.7654977 -0.3999973 0.3999973 1.3287624 0.3999973 1.3287624 0.0000028 0.0004717 0.0002728 3.3263079 0.0000066
hsa.miR.145.5p 3.405874 3.362197 0.3557557 3.294171 3.279622 0.2492612 3.517576 3.469948 0.4086525 0.2234046 -0.2234046 -0.7421341 -0.2234046 -0.7421341 0.0000030 0.0004969 0.0002844 3.3037592 0.0000069
hsa.miR.92b.3p 1.748375 1.731752 0.3770581 1.865526 1.808052 0.2617419 1.631225 1.619889 0.4352821 -0.2343014 0.2343014 0.7783325 0.2343014 0.7783325 0.0000039 0.0006400 0.0003624 3.1938498 0.0000088
hsa.miR.629.5p 1.718671 1.767577 0.2822432 1.804344 1.819456 0.2016194 1.632997 1.650836 0.3234272 -0.1713470 0.1713470 0.5692024 0.1713470 0.5692024 0.0000064 0.0010697 0.0005993 2.9707322 0.0000145
hsa.miR.99a.5p 2.916024 2.912899 0.3862114 2.800660 2.801168 0.3181101 3.031388 3.020665 0.4143220 0.2307282 -0.2307282 -0.7664624 -0.2307282 -0.7664624 0.0000086 0.0014262 0.0007904 2.8458122 0.0000190
hsa.miR.22.3p 4.901288 4.935034 0.2599745 4.979132 4.983023 0.1344526 4.823443 4.825832 0.3248074 -0.1556883 0.1556883 0.5171853 0.1556883 0.5171853 0.0000101 0.0016706 0.0009158 2.7771206 0.0000220
hsa.miR.10b.5p 4.284825 4.160990 0.6577961 4.092498 4.126032 0.2208435 4.477153 4.339191 0.8637267 0.3846552 -0.3846552 -1.2777969 -0.3846552 -1.2777969 0.0000184 0.0030501 0.0016537 2.5156825 0.0000396
hsa.miR.142.3p 2.908082 2.993345 0.5523317 3.062898 3.069081 0.3324835 2.753266 2.754739 0.6738583 -0.3096321 0.3096321 1.0285755 0.3096321 1.0285755 0.0000350 0.0058148 0.0031176 2.2354673 0.0000745
hsa.miR.16.5p 2.418426 2.396977 0.2678058 2.344887 2.354009 0.1520643 2.491965 2.532161 0.3317754 0.1470782 -0.1470782 -0.4885833 -0.1470782 -0.4885833 0.0000518 0.0086050 0.0045617 2.0652508 0.0001089
hsa.miR.10a.5p 4.453193 4.572159 0.4172756 4.567356 4.619375 0.2726925 4.339031 4.448143 0.4992066 -0.2283251 0.2283251 0.7584795 0.2283251 0.7584795 0.0000528 0.0087721 0.0045974 2.0568959 0.0001097
hsa.miR.365a.3p 1.558982 1.524607 0.3288053 1.472455 1.482687 0.2603646 1.645509 1.646930 0.3665257 0.1730535 -0.1730535 -0.5748714 -0.1730535 -0.5748714 0.0000972 0.0161318 0.0083574 1.7923179 0.0001970
hsa.miR.365b.3p 1.559072 1.524607 0.3286232 1.472600 1.482687 0.2598729 1.645545 1.646930 0.3665720 0.1729456 -0.1729456 -0.5745130 -0.1729456 -0.5745130 0.0000973 0.0161562 0.0083574 1.7916615 0.0001970
hsa.miR.20a.5p 2.167170 2.188072 0.3531756 2.258504 2.252491 0.2151305 2.075836 2.092815 0.4330362 -0.1826683 0.1826683 0.6068109 0.1826683 0.6068109 0.0001387 0.0230266 0.0116520 1.6377698 0.0002774
hsa.miR.195.5p 1.610326 1.644712 0.4593292 1.492892 1.530241 0.2351047 1.727760 1.829698 0.5838159 0.2348685 -0.2348685 -0.7802161 -0.2348685 -0.7802161 0.0001727 0.0286635 0.0143318 1.5426701 0.0003412
hsa.miR.26a.5p 3.363368 3.336961 0.2249507 3.307628 3.297116 0.1471895 3.419108 3.407502 0.2715615 0.1114806 -0.1114806 -0.3703306 -0.1114806 -0.3703306 0.0002606 0.0432635 0.0213711 1.3638783 0.0005090
hsa.miR.145.3p 1.779374 1.754841 0.2975307 1.706504 1.693139 0.2239980 1.852244 1.840403 0.3420364 0.1457395 -0.1457395 -0.4841363 -0.1457395 -0.4841363 0.0002988 0.0495960 0.0242005 1.3045533 0.0005767
hsa.miR.142.5p 1.517108 1.642632 0.7709832 1.707036 1.728336 0.3336782 1.327180 1.413437 1.0051222 -0.3798562 0.3798562 1.2618548 0.3798562 1.2618548 0.0003066 0.0509030 0.0245316 1.2932569 0.0005851
hsa.miR.155.5p 2.133480 2.195366 0.4195203 2.233687 2.297068 0.3475670 2.033272 2.006969 0.4610634 -0.2004154 0.2004154 0.6657657 0.2004154 0.6657657 0.0004167 0.0691679 0.0329172 1.1600957 0.0007860
hsa.let.7d.5p 2.204108 2.185887 0.2122742 2.154807 2.147458 0.1420691 2.253409 2.269843 0.2558420 0.0986016 -0.0986016 -0.3275473 -0.0986016 -0.3275473 0.0006292 0.1044446 0.0490764 0.9811142 0.0011735
hsa.miR.101.3p 4.064812 4.041831 0.3932389 3.975163 3.934040 0.1680979 4.154461 4.258030 0.5159943 0.1792983 -0.1792983 -0.5956161 -0.1792983 -0.5956161 0.0008471 0.1406149 0.0652250 0.8519688 0.0015624
hsa.miR.15a.5p 1.962872 1.985385 0.2869339 2.027709 2.007988 0.1768348 1.898034 1.940992 0.3545154 -0.1296752 0.1296752 0.4307717 0.1296752 0.4307717 0.0008984 0.1491266 0.0678151 0.8264448 0.0016209
hsa.miR.140.3p 3.000561 3.018035 0.2271711 2.949429 2.935298 0.2005767 3.051693 3.090265 0.2412282 0.1022638 -0.1022638 -0.3397129 -0.1022638 -0.3397129 0.0008923 0.1481224 0.0678151 0.8293794 0.0016209
hsa.miR.193a.5p 2.336431 2.300347 0.3276854 2.266929 2.278125 0.1938504 2.405932 2.378371 0.4104126 0.1390021 -0.1390021 -0.4617551 -0.1390021 -0.4617551 0.0018588 0.3085666 0.1375538 0.5106511 0.0033179
hsa.let.7b.5p 4.195831 4.209035 0.2924159 4.256938 4.282751 0.1889508 4.134724 4.104211 0.3585804 -0.1222138 0.1222138 0.4059855 0.1222138 0.4059855 0.0021512 0.3570954 0.1570359 0.4472157 0.0037989
hsa.miR.425.5p 1.923042 1.930834 0.3243793 1.987356 1.978949 0.2395357 1.858729 1.880977 0.3817182 -0.1286263 0.1286263 0.4272874 0.1286263 0.4272874 0.0035774 0.5938450 0.2575713 0.2263269 0.0062510
hsa.miR.99b.5p 4.315202 4.300285 0.2601134 4.263737 4.282048 0.1951765 4.366667 4.343604 0.3041630 0.1029303 -0.1029303 -0.3419271 -0.1029303 -0.3419271 0.0036421 0.6045962 0.2585923 0.2185346 0.0062979
hsa.miR.146b.3p 1.886946 1.965307 0.4754291 1.979342 1.993235 0.3044790 1.794549 1.860622 0.5867326 -0.1847929 0.1847929 0.6138687 0.1847929 0.6138687 0.0043674 0.7249845 0.3057163 0.1396713 0.0074741
hsa.miR.128.3p 1.955925 1.943604 0.2304255 2.000480 1.993252 0.1994596 1.911370 1.897153 0.2508004 -0.0891101 0.0891101 0.2960174 0.0891101 0.2960174 0.0044531 0.7392212 0.3072666 0.1312256 0.0075431
hsa.miR.34a.5p 2.074666 2.121651 0.2675331 2.125712 2.136585 0.1931050 2.023620 2.059319 0.3182343 -0.1020923 0.1020923 0.3391431 0.1020923 0.3391431 0.0050922 0.8453067 0.3462702 0.0729857 0.0085385
hsa.miR.1307.5p 2.032668 2.074195 0.4746939 2.121558 2.143052 0.3137240 1.943777 1.930948 0.5817384 -0.1777808 0.1777808 0.5905749 0.1777808 0.5905749 0.0060341 1.0000000 0.4042856 0.0000000 0.0100166
hsa.miR.146a.5p 1.834930 1.878293 0.5428638 1.936147 1.968467 0.3763981 1.733712 1.804289 0.6556095 -0.2024345 0.2024345 0.6724729 0.2024345 0.6724729 0.0062340 1.0000000 0.4114442 0.0000000 0.0102460
hsa.miR.660.5p 1.637156 1.706445 0.5648980 1.742335 1.736172 0.1929619 1.531977 1.656099 0.7626676 -0.2103579 0.2103579 0.6987938 0.2103579 0.6987938 0.0065781 1.0000000 0.4275745 0.0000000 0.0107055
hsa.miR.1307.3p 2.668366 2.682073 0.3103661 2.725068 2.712113 0.1957961 2.611663 2.652759 0.3856627 -0.1134058 0.1134058 0.3767259 0.1134058 0.3767259 0.0074280 1.0000000 0.4753900 0.0000000 0.0119713
hsa.miR.17.3p 2.280466 2.310070 0.2760079 2.330198 2.324300 0.1630184 2.230733 2.251456 0.3485844 -0.0994646 0.0994646 0.3304142 0.0994646 0.3304142 0.0083333 1.0000000 0.5249975 0.0000000 0.0133012
hsa.miR.103a.3p 4.078473 4.104653 0.2058716 4.115271 4.122395 0.1680536 4.041675 4.049734 0.2327882 -0.0735962 0.0735962 0.2444812 0.0735962 0.2444812 0.0086789 1.0000000 0.5380907 0.0000000 0.0137209
hsa.miR.484 1.597652 1.580193 0.2719205 1.550795 1.553409 0.1598979 1.644510 1.656175 0.3443530 0.0937157 -0.0937157 -0.3113168 -0.0937157 -0.3113168 0.0116710 1.0000000 0.7119329 0.0000000 0.0182773
hsa.miR.452.5p 1.609759 1.684291 0.4772755 1.691423 1.719895 0.2895699 1.528095 1.619481 0.6003483 -0.1633282 0.1633282 0.5425647 0.1633282 0.5425647 0.0122598 1.0000000 0.7355852 0.0000000 0.0189271
hsa.miR.584.5p 1.562512 1.609699 0.4055066 1.631695 1.617142 0.3258710 1.493330 1.570896 0.4632148 -0.1383646 0.1383646 0.4596372 0.1383646 0.4596372 0.0123140 1.0000000 0.7355852 0.0000000 0.0189271
hsa.miR.9.5p 2.452682 2.417320 0.5298502 2.364790 2.362214 0.3600516 2.540574 2.612118 0.6472233 0.1757844 -0.1757844 -0.5839430 -0.1757844 -0.5839430 0.0151165 1.0000000 0.8767577 0.0000000 0.0230215
hsa.miR.361.3p 2.114070 2.138671 0.2668813 2.157051 2.182644 0.1719882 2.071090 2.091288 0.3313815 -0.0859613 0.0859613 0.2855574 0.0859613 0.2855574 0.0184170 1.0000000 1.0000000 0.0000000 0.0277930
hsa.let.7g.5p 2.743360 2.731598 0.2633678 2.701240 2.687710 0.2007329 2.785480 2.824734 0.3090374 0.0842401 -0.0842401 -0.2798397 -0.0842401 -0.2798397 0.0191012 1.0000000 1.0000000 0.0000000 0.0285658
hsa.let.7a.3p 1.362918 1.456311 0.5840349 1.454096 1.472496 0.2172292 1.271740 1.413722 0.7883141 -0.1823566 0.1823566 0.6057754 0.1823566 0.6057754 0.0227488 1.0000000 1.0000000 0.0000000 0.0337170
hsa.miR.28.3p 3.478699 3.452430 0.2371295 3.442283 3.445475 0.2164372 3.515115 3.488616 0.2519303 0.0728325 -0.0728325 -0.2419443 -0.0728325 -0.2419443 0.0243409 1.0000000 1.0000000 0.0000000 0.0355939
hsa.miR.23b.3p 3.145549 3.139008 0.1654088 3.120156 3.106189 0.1396735 3.170942 3.185124 0.1848453 0.0507862 -0.0507862 -0.1687082 -0.0507862 -0.1687082 0.0244440 1.0000000 1.0000000 0.0000000 0.0355939
hsa.miR.29b.3p 2.730065 2.739194 0.3702233 2.785646 2.771371 0.2357865 2.674484 2.713670 0.4621597 -0.1111620 0.1111620 0.3692721 0.1111620 0.3692721 0.0281053 1.0000000 1.0000000 0.0000000 0.0405694

We may want to futher visualize the results of differential expression using heatmap and vulcano plot.

ks.heatmap(x = dplyr::select(trainx, sig_de$miR), rlab = data.frame(Class = train$Class), zscore = F, margins = c(10, 10))

Z-scoring the values before clustering and plotting and help to gain more insights.

ks.heatmap(x = dplyr::select(trainx, sig_de$miR), rlab = data.frame(Class = train$Class), zscore = T, margins = c(10, 10))

Let’s plot the vulcano plot and label top 10 most significant miRNAs:

ks.vulcano_plot(selected_miRNAs = de$miR, DE = de, only_label = sig_de$miR[1:10])

We may also what to check the consistency of differential expression between datasets:

de_test = ks.miRNA_differential_expression(dplyr::select(test, starts_with("hsa")), test$Class)
de_valid = ks.miRNA_differential_expression(dplyr::select(valid, starts_with("hsa")), valid$Class)
ks.correlation_plot(de$log2FC, de_test$log2FC, "log2FC on training set", "log2FC on test set", "", yx = T)

ks.correlation_plot(de$log2FC, de_valid$log2FC, "log2FC on training set", "log2FC on validation set", "", yx = T)

ks.correlation_plot(de_test$log2FC, de_valid$log2FC, "log2FC on test set", "log2FC on validation set", "", yx = T)

miRNA selection

The main feature of this package is the shotgun-like feature selection evaluation of possible miRNA signatures of biological processes. The function can be applied in a straightforward way, e.g.:

selected_features = ks.miRNAselector(wd = getwd(), m = 1:70, max_iterations = 10, stamp = "tutorial")

But, for largers projects we suggest using the following wrapper wich will perform the feature selection in parallel, significantly reducing computational time. We do not recommend using more than 5 threads, beacuse some of the methods inhereditly use multicore processing.

readLines("Tutorial_miRNAselector.R") %>% paste0(collapse = "\n") %>% cat
#> #options(warn = -1)
#> suppressMessages(library(foreach))
#> suppressMessages(library(doParallel))
#> suppressMessages(library(parallel))
#> suppressMessages(library(doSNOW))
#> 
#> m = 1:56 # which methods to check?
#> 
#> cl <- makeCluster(5) # 5 threds by default
#> doSNOW::registerDoSNOW(cl)
#> iterations = length(m)
#> pb <- txtProgressBar(max = iterations, style = 3)
#> progress <- function(n) setTxtProgressBar(pb, n)
#> opts <- list(progress = progress)
#> foreach(i = m, .options.snow = opts) %dopar%
#> {
#>   suppressMessages(library(miRNAselector))
#>   setwd("~/public/Projekty/KS/miRNAselector/vignettes") # change it you to your working directory
#>   ks.miRNAselector(m = i, max_iterations = 1, stamp = "tutorial", debug = F, # we set debug to false (to make the package smaller), you may also want to change stamp to something meaningful, max_iterations was set to 1 to recude the computational time.. in real life scenarios it is resonable to use at least 10 iterations.
#>                   prefer_no_features = 11, # Few methods are filter rather than wrapper methods, thus requires the maximum number of maximum features.
#>                   conda_path = "/home/konrad/anaconda3/bin/conda", # Methods line WxNet requires usage of python. In setup script we create conda enviorment. Providing conda_path makes it easier to activate env. We prefer this apporach over use_condaenv.
#>                   timeout = 600) # We don't want to wait eternity in this tutorial, just 10 minutes. Timeout is useful for complicated methods. Depending on your CPU 2 days may be reasonable for larger projects.
#>                    }
#> 
#> stopCluster(cl)
#> #options(warn = 0)

Few notes about what is does:

Files created for each method (e.g. for stamp=tutorial and m=1):

Pearls about the methods:

Notes about methods:

– TO DO –

The miRNA.selector functions saves all output files to temp/ directory. As users may want to run multiple selection methods in different configurations we do not recommend using the return of this function in the next steps. Instead, we provide ks.merge_formulas() which conviniently summerizes the results of feature selection. We do:

selected_sets_of_miRNAs = ks.merge_formulas(max_miRNAs = 11)  # we filter out sets with more than 11 miRNAs.
selected_sets_of_miRNAs_with_own = ks.merge_formulas(max_miRNAs = 11, add = list(my_own_signature = c("hsa.miR.192.5p", "hsa.let.7g.5p", "hsa.let.7a.5p", 
    "hsa.let.7d.5p", "hsa.miR.194.5p", "hsa.miR.98.5p", "hsa.let.7f.5p", "hsa.miR.26b.5p")))  # you can also add your own signature (for example selected from literature)

Note that:

Let’s analyze the process of feature selection:

all_sets = readRDS("featureselection_formulas_all.RDS")
length(all_sets)  # How many feature selection methods completed in time?
#> [1] 47
final_sets = readRDS("featureselection_formulas_final.RDS")
length(final_sets)  # How many feature selection methods completed in time and fulfilled max_miRNA criteria? (remember about fcsig and cfs_sig)
#> [1] 33
featureselection_formulas_final = fread("featureselection_formulas_final.csv")
ks.table(featureselection_formulas_final)  # show information about selected formulas
name formula ile_miRNA
cfs_sig Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.338.3p + hsa.miR.199b.5p + hsa.miR.381.3p + hsa.miR.199a.3p + hsa.miR.194.5p + hsa.miR.134.5p + hsa.miR.192.5p + hsa.miR.200a.3p + hsa.miR.181c.3p + hsa.miR.29a.3p + hsa.miR.378a.3p + hsa.miR.144.5p + hsa.miR.335.3p + hsa.miR.126.5p + hsa.let.7c.5p + hsa.miR.378a.5p + hsa.miR.181c.5p + hsa.miR.126.3p + hsa.miR.185.5p + hsa.miR.29c.3p + hsa.miR.151a.5p + hsa.miR.92b.3p + hsa.miR.10b.5p + hsa.miR.195.5p + hsa.miR.101.3p + hsa.miR.484 + hsa.miR.28.3p 34
fcsig Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p + hsa.miR.127.5p + hsa.miR.381.3p + hsa.miR.127.3p + hsa.miR.199a.3p + hsa.miR.181b.5p + hsa.miR.199b.3p + hsa.miR.194.5p + hsa.miR.134.5p + hsa.miR.30a.5p + hsa.miR.200a.5p + hsa.miR.136.5p + hsa.miR.192.5p + hsa.miR.451a + hsa.miR.200a.3p + hsa.miR.141.3p + hsa.miR.200b.3p + hsa.miR.708.3p + hsa.miR.210.3p + hsa.miR.486.5p + hsa.miR.378a.3p + hsa.miR.379.5p + hsa.miR.141.5p + hsa.miR.21.3p + hsa.miR.337.3p + hsa.miR.144.5p + hsa.miR.429 + hsa.miR.335.3p + hsa.miR.126.5p + hsa.miR.181a.3p + hsa.miR.181a.5p + hsa.miR.378a.5p + hsa.miR.146b.5p + hsa.miR.182.5p + hsa.miR.183.5p + hsa.miR.223.3p + hsa.miR.200c.3p + hsa.miR.10b.5p + hsa.miR.142.3p + hsa.miR.142.5p 50
my_own_signature Class ~ hsa.miR.192.5p + hsa.let.7g.5p + hsa.let.7a.5p + hsa.let.7d.5p + hsa.miR.194.5p + hsa.miR.98.5p + hsa.let.7f.5p + hsa.miR.26b.5p 0
bounceR-stability Class ~ hsa.miR.500a.3p + hsa.miR.126.3p + hsa.miR.125a.5p + hsa.miR.181a.2.3p + hsa.miR.1307.3p + hsa.miR.25.3p + hsa.miR.200b.3p + hsa.miR.22.3p + hsa.miR.106b.3p + hsa.miR.99a.5p + hsa.miR.452.5p 11
fcfsSMOTE Class ~ hsa.miR.21.5p + hsa.miR.30a.3p 2
fcfsSMOTE_sig Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.136.5p 4
fwrap Class ~ hsa.miR.21.5p + hsa.miR.126.5p + hsa.miR.375.3p 3
fwrap_sig Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.375.3p 3
AUC_MDL Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p 11
SU_MDL Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.30a.3p + hsa.miR.10b.5p + hsa.miR.126.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.126.3p 11
CorrSF_MDL Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.194.5p + hsa.miR.10b.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.338.3p + hsa.miR.217.5p + hsa.miR.378a.5p + hsa.miR.195.5p 11
sigtop Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p 11
sigtopBonf Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p 11
sigtopHolm Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p 11
topFC Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.200a.3p + hsa.miR.708.3p + hsa.miR.144.5p + hsa.miR.141.3p 11
sigtopSMOTE Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p 11
sigtopBonfSMOTE Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p 11
sigtopHolmSMOTE Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p 11
topFCSMOTE Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p 11
AUC_MDLSMOTE Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.200a.5p + hsa.miR.127.5p 11
SU_MDLSMOTE Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.708.3p + hsa.miR.375.3p + hsa.miR.30c.2.3p + hsa.miR.217.5p + hsa.miR.194.5p + hsa.miR.200a.3p + hsa.miR.132.3p + hsa.miR.199a.5p 11
CorrSF_MDLSMOTE Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.708.3p + hsa.miR.30a.3p + hsa.miR.136.5p + hsa.miR.132.3p 6
AUC_MDL_sig Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p 11
SU_MDL_sig Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.30a.3p + hsa.miR.10b.5p + hsa.miR.126.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.126.3p 11
CorrSF_MDL_sig Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.194.5p + hsa.miR.10b.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.338.3p + hsa.miR.217.5p + hsa.miR.378a.5p + hsa.miR.195.5p 11
AUC_MDLSMOTE_sig Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.127.5p + hsa.miR.139.5p 11
SU_MDLSMOTE_sig Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.375.3p + hsa.miR.708.3p + hsa.miR.136.5p + hsa.miR.30c.2.3p + hsa.miR.194.5p + hsa.miR.200a.3p + hsa.miR.217.5p + hsa.miR.132.3p 11
CorrSF_MDLSMOTE_sig Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30a.3p 3
RandomForestRFE Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.375.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.217.5p + hsa.miR.194.5p + hsa.miR.192.5p + hsa.miR.126.5p 10
cfsSMOTE Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.132.3p + hsa.miR.136.5p + hsa.miR.30e.3p 5
cfsSMOTE_sig Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.132.3p + hsa.miR.136.5p 5
Mystepwise_glm_binomial Class ~ hsa.miR.21.5p + hsa.miR.130a.3p + hsa.miR.30e.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.146b.5p 6
Mystepwise_sig_glm_binomial Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p 6

Note that my_own_signture has 0 miRNAs according to the table. This trick is done to make sure that every sigure added manually will survive filtering.

Summary:

hist(featureselection_formulas_final$ile_miRNA[-which(featureselection_formulas_final$ile_miRNA == 0)], breaks = ncol(train))  # Histogram showing how many miRNAs were selected in final set.

psych::describe(featureselection_formulas_final$ile_miRNA[-which(featureselection_formulas_final$ile_miRNA == 0)])  # Descriptive statistics of how many features where selected in the final set.
#>    vars  n  mean   sd median trimmed mad min max range skew kurtosis   se
#> X1    1 32 10.81 9.01     11    9.35   0   2  50    48 2.96     9.64 1.59

Benchmarking

In the next step of looking for the best signature, we perform benchmarking. This tests all the signatures using different data mining methods. Here is the example of benchmark with default parameters:

readLines("Tutorial_benchmark.R") %>% paste0(collapse = "\n") %>% cat
#> suppressMessages(library(miRNAselector))
#> ks.benchmark(search_iters = 5, # 5 random hyperparameter sets will be checked; 5 is set here for speed purposes.. for real projects use more
#>              algorithms = c("mlp", "mlpML", "svmRadial", "svmLinear", "rf", "C5.0", "rpart",
#>                             "rpart2", "ctree"), # default set of methods, note that logistic regression (glm) is always included
#>              output_file = paste0("benchmark.csv")) # the main output

As benchmarking is done the main result file is saved as declared in output_file parameter. This file contains the performance metrics of the selected signiture across different methods of data mining modelling. Let’s take a look:

ks.table(fread("benchmark.csv"))
V1 method SMOTE miRy glm_modelname glm_train_ROCAUC glm_train_ROCAUC_lower95CI glm_train_ROCAUC_upper95CI glm_train_Accuracy glm_train_Sensitivity glm_train_Specificity glm_test_Accuracy glm_test_Sensitivity glm_test_Specificity glm_valid_Accuracy glm_valid_Sensitivity glm_valid_Specificity mlp_modelname mlp_train_ROCAUC mlp_train_ROCAUC_lower95CI mlp_train_ROCAUC_upper95CI mlp_train_Accuracy mlp_train_Sensitivity mlp_train_Specificity mlp_test_Accuracy mlp_test_Sensitivity mlp_test_Specificity mlp_valid_Accuracy mlp_valid_Sensitivity mlp_valid_Specificity mlpML_modelname mlpML_train_ROCAUC mlpML_train_ROCAUC_lower95CI mlpML_train_ROCAUC_upper95CI mlpML_train_Accuracy mlpML_train_Sensitivity mlpML_train_Specificity mlpML_test_Accuracy mlpML_test_Sensitivity mlpML_test_Specificity mlpML_valid_Accuracy mlpML_valid_Sensitivity mlpML_valid_Specificity svmRadial_modelname svmRadial_train_ROCAUC svmRadial_train_ROCAUC_lower95CI svmRadial_train_ROCAUC_upper95CI svmRadial_train_Accuracy svmRadial_train_Sensitivity svmRadial_train_Specificity svmRadial_test_Accuracy svmRadial_test_Sensitivity svmRadial_test_Specificity svmRadial_valid_Accuracy svmRadial_valid_Sensitivity svmRadial_valid_Specificity svmLinear_modelname svmLinear_train_ROCAUC svmLinear_train_ROCAUC_lower95CI svmLinear_train_ROCAUC_upper95CI svmLinear_train_Accuracy svmLinear_train_Sensitivity svmLinear_train_Specificity svmLinear_test_Accuracy svmLinear_test_Sensitivity svmLinear_test_Specificity svmLinear_valid_Accuracy svmLinear_valid_Sensitivity svmLinear_valid_Specificity rf_modelname rf_train_ROCAUC rf_train_ROCAUC_lower95CI rf_train_ROCAUC_upper95CI rf_train_Accuracy rf_train_Sensitivity rf_train_Specificity rf_test_Accuracy rf_test_Sensitivity rf_test_Specificity rf_valid_Accuracy rf_valid_Sensitivity rf_valid_Specificity C5.0_modelname C5.0_train_ROCAUC C5.0_train_ROCAUC_lower95CI C5.0_train_ROCAUC_upper95CI C5.0_train_Accuracy C5.0_train_Sensitivity C5.0_train_Specificity C5.0_test_Accuracy C5.0_test_Sensitivity C5.0_test_Specificity C5.0_valid_Accuracy C5.0_valid_Sensitivity C5.0_valid_Specificity rpart_modelname rpart_train_ROCAUC rpart_train_ROCAUC_lower95CI rpart_train_ROCAUC_upper95CI rpart_train_Accuracy rpart_train_Sensitivity rpart_train_Specificity rpart_test_Accuracy rpart_test_Sensitivity rpart_test_Specificity rpart_valid_Accuracy rpart_valid_Sensitivity rpart_valid_Specificity rpart2_modelname rpart2_train_ROCAUC rpart2_train_ROCAUC_lower95CI rpart2_train_ROCAUC_upper95CI rpart2_train_Accuracy rpart2_train_Sensitivity rpart2_train_Specificity rpart2_test_Accuracy rpart2_test_Sensitivity rpart2_test_Specificity rpart2_valid_Accuracy rpart2_valid_Sensitivity rpart2_valid_Specificity ctree_modelname ctree_train_ROCAUC ctree_train_ROCAUC_lower95CI ctree_train_ROCAUC_upper95CI ctree_train_Accuracy ctree_train_Sensitivity ctree_train_Specificity ctree_test_Accuracy ctree_test_Sensitivity ctree_test_Specificity ctree_valid_Accuracy ctree_valid_Sensitivity ctree_valid_Specificity
1 cfs_sig No Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.338.3p + hsa.miR.199b.5p + hsa.miR.381.3p + hsa.miR.199a.3p + hsa.miR.194.5p + hsa.miR.134.5p + hsa.miR.192.5p + hsa.miR.200a.3p + hsa.miR.181c.3p + hsa.miR.29a.3p + hsa.miR.378a.3p + hsa.miR.144.5p + hsa.miR.335.3p + hsa.miR.126.5p + hsa.let.7c.5p + hsa.miR.378a.5p + hsa.miR.181c.5p + hsa.miR.126.3p + hsa.miR.185.5p + hsa.miR.29c.3p + hsa.miR.151a.5p + hsa.miR.92b.3p + hsa.miR.10b.5p + hsa.miR.195.5p + hsa.miR.101.3p + hsa.miR.484 + hsa.miR.28.3p 1584462908 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.8888889 0.9166667 0.8611111 1.0000000 1.0000000 1.0000000 1584463328 0.9974670 0.9934914 1.0000000 0.9906542 1.0000000 0.9813084 0.9722222 1.0000000 0.9444444 0.9857143 1.0000000 0.9714286 1584463844 0.9963316 0.9910107 1.0000000 0.9906542 1.0000000 0.9813084 0.9722222 1.0000000 0.9444444 0.9857143 1.0000000 0.9714286 1584464368 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584465010 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9583333 0.9722222 0.9444444 0.9714286 1.0000000 0.9428571 1584465465 1 1 1 1 1 1 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584466016 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9861111 1.0000000 0.9722222 0.9857143 1.0000000 0.9714286 1584467102 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467489 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467877 0.9768539 0.9567499 0.9969579 0.9485981 0.9813084 0.9158879 0.9305556 1.0000000 0.8611111 0.9571429 1.0000000 0.9142857
2 fcsig No Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p + hsa.miR.127.5p + hsa.miR.381.3p + hsa.miR.127.3p + hsa.miR.199a.3p + hsa.miR.181b.5p + hsa.miR.199b.3p + hsa.miR.194.5p + hsa.miR.134.5p + hsa.miR.30a.5p + hsa.miR.200a.5p + hsa.miR.136.5p + hsa.miR.192.5p + hsa.miR.451a + hsa.miR.200a.3p + hsa.miR.141.3p + hsa.miR.200b.3p + hsa.miR.708.3p + hsa.miR.210.3p + hsa.miR.486.5p + hsa.miR.378a.3p + hsa.miR.379.5p + hsa.miR.141.5p + hsa.miR.21.3p + hsa.miR.337.3p + hsa.miR.144.5p + hsa.miR.429 + hsa.miR.335.3p + hsa.miR.126.5p + hsa.miR.181a.3p + hsa.miR.181a.5p + hsa.miR.378a.5p + hsa.miR.146b.5p + hsa.miR.182.5p + hsa.miR.183.5p + hsa.miR.223.3p + hsa.miR.200c.3p + hsa.miR.10b.5p + hsa.miR.142.3p + hsa.miR.142.5p 1584462916 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9166667 0.9166667 0.9166667 0.9571429 1.0000000 0.9142857 1584463336 0.5373832 0.5123484 0.5624180 0.5000000 0.0000000 1.0000000 0.5000000 0.0000000 1.0000000 0.5000000 0.0000000 1.0000000 1584463852 0.9979037 0.9948019 1.0000000 0.9906542 1.0000000 0.9813084 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584464376 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584465017 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9305556 1.0000000 0.8611111 0.9857143 1.0000000 0.9714286 1584465473 1 1 1 1 1 1 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584466026 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9861111 1.0000000 0.9722222 0.9857143 1.0000000 0.9714286 1584467111 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467497 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467885 0.9943663 0.9892504 0.9994823 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
3 my_own_signature No Class ~ hsa.miR.192.5p + hsa.let.7g.5p + hsa.let.7a.5p + hsa.let.7d.5p + hsa.miR.194.5p + hsa.miR.98.5p + hsa.let.7f.5p + hsa.miR.26b.5p 1584462924 0.9239235 0.8901386 0.9577083 0.8317757 0.8504673 0.8130841 0.8472222 0.8055556 0.8888889 0.8857143 0.8857143 0.8857143 1584463344 0.9246222 0.8917772 0.9574673 0.8130841 0.9813084 0.6448598 0.7638889 0.9722222 0.5555556 0.8428571 1.0000000 0.6857143 1584463860 0.9363263 0.9041742 0.9684784 0.8411215 0.9626168 0.7196262 0.8055556 0.9722222 0.6388889 0.8857143 1.0000000 0.7714286 1584464384 0.9765918 0.9565725 0.9966111 0.9299065 0.9719626 0.8878505 0.8750000 0.8888889 0.8611111 0.9714286 0.9714286 0.9714286 1584465025 0.9195563 0.8844029 0.9547097 0.8317757 0.8504673 0.8130841 0.8194444 0.8055556 0.8333333 0.8571429 0.8857143 0.8285714 1584465481 1 1 1 1 1 1 0.8611111 0.8611111 0.8611111 0.9428571 0.9714286 0.9142857 1584466034 0.9985152 0.9963905 1.0000000 0.9719626 0.9813084 0.9626168 0.8472222 0.8055556 0.8888889 0.9142857 0.8857143 0.9428571 1584467119 0.9508254 0.9228791 0.9787717 0.9018692 0.8504673 0.9532710 0.7638889 0.6944444 0.8333333 0.8571429 0.7714286 0.9428571 1584467505 0.9021312 0.8620675 0.9421949 0.8831776 0.9065421 0.8598131 0.7500000 0.7222222 0.7777778 0.9285714 0.9428571 0.9142857 1584467893 0.9753254 0.9605664 0.9900843 0.8971963 0.8317757 0.9626168 0.7916667 0.6944444 0.8888889 0.8714286 0.8000000 0.9428571
4 bounceR.stability No Class ~ hsa.miR.500a.3p + hsa.miR.126.3p + hsa.miR.125a.5p + hsa.miR.181a.2.3p + hsa.miR.1307.3p + hsa.miR.25.3p + hsa.miR.200b.3p + hsa.miR.22.3p + hsa.miR.106b.3p + hsa.miR.99a.5p + hsa.miR.452.5p 1584462932 0.9645384 0.9421954 0.9868814 0.9018692 0.9158879 0.8878505 0.8472222 0.8333333 0.8611111 0.9428571 1.0000000 0.8857143 1584463352 0.9375491 0.9027342 0.9723640 0.8738318 0.9626168 0.7850467 0.9027778 0.9722222 0.8333333 0.9142857 1.0000000 0.8285714 1584463868 0.9289021 0.8945307 0.9632734 0.8457944 0.8317757 0.8598131 0.8611111 0.8333333 0.8888889 0.9000000 0.9142857 0.8857143 1584464392 0.9996506 0.9988872 1.0000000 0.9906542 1.0000000 0.9813084 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584465033 0.9626168 0.9396477 0.9855859 0.8971963 0.9158879 0.8785047 0.8750000 0.9166667 0.8333333 0.9428571 0.9714286 0.9142857 1584465489 1 1 1 1 1 1 0.9166667 0.9444444 0.8888889 0.9857143 1.0000000 0.9714286 1584466043 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.8750000 0.8888889 0.8611111 0.9571429 0.9714286 0.9428571 1584467127 0.9646257 0.9402313 0.9890202 0.9392523 0.9532710 0.9252336 0.8611111 0.8888889 0.8333333 0.9142857 0.9714286 0.8571429 1584467513 0.9646257 0.9402313 0.9890202 0.9392523 0.9532710 0.9252336 0.8611111 0.8888889 0.8333333 0.9142857 0.9714286 0.8571429 1584467901 0.9828806 0.9706308 0.9951304 0.9252336 0.9345794 0.9158879 0.8611111 0.8888889 0.8333333 0.9142857 1.0000000 0.8285714
5 fcfsSMOTE Yes Class ~ hsa.miR.21.5p + hsa.miR.30a.3p 1584462940 0.9907048 0.9895335 0.9918762 0.9694983 0.9733645 0.9656704 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584463372 0.9904611 0.9892241 0.9916980 0.9768447 0.9824299 0.9713149 0.9305556 0.9722222 0.8888889 0.9857143 1.0000000 0.9714286 1584463887 0.9904518 0.9892245 0.9916792 0.9642907 0.9541121 0.9743685 0.9027778 0.9166667 0.8888889 0.9857143 1.0000000 0.9714286 1584464430 0.9899233 0.9882711 0.9915755 0.9866090 1.0000000 0.9733506 0.8750000 0.8888889 0.8611111 0.9285714 0.9428571 0.9142857 1584465048 0.9906759 0.9895011 0.9918506 0.9738690 0.9819626 0.9658555 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584465506 1 1 1 1 1 1 0.9027778 0.8055556 1.0000000 0.8285714 0.6857143 0.9714286 1584466097 0.9989302 0.9986219 0.9992384 0.9969312 1.0000000 0.9938928 0.9305556 0.8888889 0.9722222 0.9000000 0.8285714 0.9714286 1584467136 0.9729217 0.9708267 0.9750166 0.9712652 0.9913084 0.9514204 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584467522 0.9729217 0.9708267 0.9750166 0.9712652 0.9913084 0.9514204 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584467909 0.9993709 0.9991560 0.9995858 0.9973497 1.0000000 0.9947256 0.9166667 0.8611111 0.9722222 0.9571429 0.9428571 0.9714286
6 fcfsSMOTE_sig Yes Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.136.5p 1584462957 0.9965117 0.9958363 0.9971871 0.9800995 0.9822430 0.9779772 0.9583333 0.9722222 0.9444444 0.9714286 1.0000000 0.9428571 1584463397 0.9953074 0.9944878 0.9961270 0.9842842 0.9822430 0.9863052 0.9444444 0.9722222 0.9166667 0.9714286 1.0000000 0.9428571 1584463912 0.9955242 0.9947351 0.9963132 0.9790766 0.9913084 0.9669659 0.9444444 1.0000000 0.8888889 0.9571429 1.0000000 0.9142857 1584464464 0.9977802 0.9971010 0.9984594 0.9940020 1.0000000 0.9880633 0.9305556 0.9166667 0.9444444 0.9857143 1.0000000 0.9714286 1584465068 0.9958849 0.9951245 0.9966454 0.9841447 0.9822430 0.9860276 0.9444444 0.9722222 0.9166667 0.9714286 1.0000000 0.9428571 1584465529 1 1 1 1 1 1 0.8888889 0.8333333 0.9444444 0.9571429 0.9142857 1.0000000 1584466155 1.0000000 1.0000000 1.0000000 0.9995815 1.0000000 0.9991672 0.9305556 0.9166667 0.9444444 0.9571429 0.9428571 0.9714286 1584467150 0.9729217 0.9708267 0.9750166 0.9712652 0.9913084 0.9514204 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584467536 0.9866442 0.9851528 0.9881355 0.9858651 0.9913084 0.9804756 0.9305556 0.9444444 0.9166667 0.9857143 1.0000000 0.9714286 1584467926 0.9996475 0.9994508 0.9998442 0.9980936 1.0000000 0.9962062 0.9305556 0.9166667 0.9444444 0.9285714 0.8857143 0.9714286
7 fwrap No Class ~ hsa.miR.21.5p + hsa.miR.126.5p + hsa.miR.375.3p 1584462973 0.9888200 0.9789249 0.9987150 0.9485981 0.9439252 0.9532710 0.9027778 0.9444444 0.8611111 0.9857143 1.0000000 0.9714286 1584463409 0.9765918 0.9608023 0.9923814 0.9252336 0.9626168 0.8878505 0.8333333 0.9722222 0.6944444 0.9428571 0.9714286 0.9142857 1584463924 0.9791248 0.9645951 0.9936545 0.9252336 0.9719626 0.8785047 0.8194444 0.9722222 0.6666667 0.9142857 1.0000000 0.8285714 1584464481 0.9902175 0.9749209 1.0000000 0.9813084 1.0000000 0.9626168 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584465083 0.9886453 0.9786723 0.9986183 0.9485981 0.9439252 0.9532710 0.9305556 0.9722222 0.8888889 0.9857143 1.0000000 0.9714286 1584465541 1 1 1 1 1 1 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584466167 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584467164 0.9863307 0.9714585 1.0000000 0.9766355 0.9813084 0.9719626 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584467550 0.9863307 0.9714585 1.0000000 0.9766355 0.9813084 0.9719626 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584467942 0.9863307 0.9714585 1.0000000 0.9766355 0.9813084 0.9719626 0.9444444 0.9722222 0.9166667 0.9714286 1.0000000 0.9428571
8 fwrap_sig No Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.375.3p 1584462981 0.9924011 0.9795847 1.0000000 0.9859813 0.9906542 0.9813084 0.9166667 0.9444444 0.8888889 0.9857143 1.0000000 0.9714286 1584463417 0.9924011 0.9800430 1.0000000 0.9766355 0.9906542 0.9626168 0.9166667 0.9722222 0.8611111 0.9857143 1.0000000 0.9714286 1584463932 0.9924011 0.9800430 1.0000000 0.9859813 0.9906542 0.9813084 0.9027778 0.9444444 0.8611111 0.9857143 1.0000000 0.9714286 1584464489 0.9956328 0.9883696 1.0000000 0.9906542 1.0000000 0.9813084 0.9305556 0.9722222 0.8888889 0.9857143 1.0000000 0.9714286 1584465091 0.9925758 0.9797890 1.0000000 0.9813084 0.9813084 0.9813084 0.9166667 0.9444444 0.8888889 0.9857143 1.0000000 0.9714286 1584465549 1 1 1 1 1 1 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584466174 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584467172 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467558 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467950 0.9906979 0.9797015 1.0000000 0.9579439 0.9345794 0.9813084 0.9166667 0.9166667 0.9166667 0.9428571 0.9714286 0.9142857
9 AUC_MDL No Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p 1584462988 0.9931872 0.9810931 1.0000000 0.9906542 1.0000000 0.9813084 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584463425 0.9904795 0.9759992 1.0000000 0.9766355 0.9719626 0.9813084 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584463940 0.9897808 0.9746001 1.0000000 0.9859813 0.9906542 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464498 0.9910036 0.9740380 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584465098 0.9894314 0.9729248 1.0000000 0.9906542 1.0000000 0.9813084 0.9444444 0.9722222 0.9166667 1.0000000 1.0000000 1.0000000 1584465557 1 1 1 1 1 1 0.9722222 1.0000000 0.9444444 0.9857143 1.0000000 0.9714286 1584466183 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584467180 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467566 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467958 0.9941480 0.9887442 0.9995518 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
10 SU_MDL No Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.30a.3p + hsa.miR.10b.5p + hsa.miR.126.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.126.3p 1584462996 0.9965936 0.9915862 1.0000000 0.9766355 0.9813084 0.9719626 0.9305556 0.9444444 0.9166667 0.9714286 1.0000000 0.9428571 1584463433 0.9927505 0.9813321 1.0000000 0.9813084 0.9906542 0.9719626 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584463948 0.9930125 0.9822433 1.0000000 0.9813084 0.9906542 0.9719626 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584464506 0.9969430 0.9910570 1.0000000 0.9906542 1.0000000 0.9813084 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584465106 0.9937986 0.9837316 1.0000000 0.9813084 0.9906542 0.9719626 0.9444444 0.9722222 0.9166667 0.9714286 1.0000000 0.9428571 1584465565 1 1 1 1 1 1 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584466190 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9722222 1.0000000 0.9444444 0.9571429 0.9714286 0.9428571 1584467188 0.9889073 0.9755302 1.0000000 0.9672897 0.9439252 0.9906542 0.9305556 0.9444444 0.9166667 0.9714286 0.9714286 0.9714286 1584467573 0.9889073 0.9755302 1.0000000 0.9672897 0.9439252 0.9906542 0.9305556 0.9444444 0.9166667 0.9714286 0.9714286 0.9714286 1584467965 0.9825749 0.9652791 0.9998707 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
11 CorrSF_MDL No Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.194.5p + hsa.miR.10b.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.338.3p + hsa.miR.217.5p + hsa.miR.378a.5p + hsa.miR.195.5p 1584463004 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584463441 0.9966809 0.9911475 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584463956 0.9964189 0.9905142 1.0000000 0.9766355 0.9719626 0.9813084 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584464514 0.9998253 0.9994066 1.0000000 0.9906542 1.0000000 0.9813084 0.9444444 0.9722222 0.9166667 1.0000000 1.0000000 1.0000000 1584465114 0.9976417 0.9931160 1.0000000 0.9906542 1.0000000 0.9813084 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584465573 1 1 1 1 1 1 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584466198 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9722222 1.0000000 0.9444444 0.9571429 0.9714286 0.9428571 1584467196 0.9889073 0.9755302 1.0000000 0.9672897 0.9439252 0.9906542 0.9166667 0.9166667 0.9166667 0.9571429 0.9714286 0.9428571 1584467581 0.9889073 0.9755302 1.0000000 0.9672897 0.9439252 0.9906542 0.9166667 0.9166667 0.9166667 0.9571429 0.9714286 0.9428571 1584467973 0.9942790 0.9890146 0.9995433 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
12 sigtop No Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p 1584463012 0.9961569 0.9902191 1.0000000 0.9859813 1.0000000 0.9719626 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584463449 0.9912219 0.9784013 1.0000000 0.9813084 0.9813084 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584463964 0.9929251 0.9817767 1.0000000 0.9813084 0.9813084 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464522 0.9939733 0.9822990 1.0000000 0.9906542 1.0000000 0.9813084 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584465122 0.9926631 0.9806675 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584465580 1 1 1 1 1 1 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584466206 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584467204 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467589 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467981 0.9905232 0.9794706 1.0000000 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
13 sigtopBonf No Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p 1584463020 0.9961569 0.9902191 1.0000000 0.9859813 1.0000000 0.9719626 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584463456 0.9913530 0.9788234 1.0000000 0.9719626 0.9626168 0.9813084 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584463972 0.9903922 0.9759600 1.0000000 0.9813084 1.0000000 0.9626168 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464530 0.9942353 0.9830720 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584465129 0.9926631 0.9806675 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584465588 1 1 1 1 1 1 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584466215 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584467212 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467597 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467989 0.9905232 0.9794706 1.0000000 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
14 sigtopHolm No Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p 1584463027 0.9961569 0.9902191 1.0000000 0.9859813 1.0000000 0.9719626 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584463464 0.9917023 0.9792337 1.0000000 0.9719626 0.9626168 0.9813084 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584463979 0.9914403 0.9788481 1.0000000 0.9813084 0.9813084 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464538 0.9939733 0.9822990 1.0000000 0.9906542 1.0000000 0.9813084 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584465137 0.9926631 0.9806675 1.0000000 0.9859813 0.9906542 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584465596 1 1 1 1 1 1 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584466223 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584467220 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467605 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467996 0.9905232 0.9794706 1.0000000 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
15 topFC No Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.200a.3p + hsa.miR.708.3p + hsa.miR.144.5p + hsa.miR.141.3p 1584463035 0.9939733 0.9856621 1.0000000 0.9813084 0.9906542 0.9719626 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584463472 0.9910909 0.9784150 1.0000000 0.9859813 1.0000000 0.9719626 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584463987 0.9916150 0.9800129 1.0000000 0.9859813 1.0000000 0.9719626 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584464546 0.9942353 0.9868483 1.0000000 0.9766355 0.9813084 0.9719626 0.9305556 0.9722222 0.8888889 0.9857143 1.0000000 0.9714286 1584465145 0.9939733 0.9853614 1.0000000 0.9813084 0.9906542 0.9719626 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584465604 1 1 1 1 1 1 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584466230 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9583333 1.0000000 0.9166667 0.9714286 1.0000000 0.9428571 1584467228 0.9816578 0.9643889 0.9989267 0.9672897 0.9719626 0.9626168 0.9583333 0.9722222 0.9444444 0.9714286 0.9714286 0.9714286 1584467613 0.9654118 0.9421733 0.9886504 0.9626168 1.0000000 0.9252336 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584468005 0.9944537 0.9893380 0.9995693 0.9579439 0.9906542 0.9252336 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571
16 sigtopSMOTE Yes Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p 1584463045 0.9956204 0.9951104 0.9961305 0.9787976 0.9814953 0.9761266 0.8888889 0.9166667 0.8611111 0.9714286 1.0000000 0.9428571 1584463494 0.9986075 0.9983098 0.9989052 0.9928860 1.0000000 0.9858425 0.8888889 0.8888889 0.8888889 0.9714286 1.0000000 0.9428571 1584464009 0.9976486 0.9971144 0.9981827 0.9883759 0.9908411 0.9859350 0.8750000 0.8888889 0.8611111 0.9714286 1.0000000 0.9428571 1584464572 1.0000000 1.0000000 1.0000000 0.9981866 1.0000000 0.9963912 0.9166667 0.8611111 0.9722222 0.9857143 0.9714286 1.0000000 1584465165 0.9948269 0.9941933 0.9954605 0.9790301 0.9814953 0.9765892 0.8888889 0.9166667 0.8611111 0.9714286 1.0000000 0.9428571 1584465634 1 1 1 1 1 1 0.8472222 0.7222222 0.9722222 0.9571429 0.9142857 1.0000000 1584466293 1.0000000 1.0000000 1.0000000 0.9999535 1.0000000 0.9999075 0.9166667 0.8611111 0.9722222 0.9857143 0.9714286 1.0000000 1584467238 0.9779993 0.9760129 0.9799858 0.9682429 0.9816822 0.9549366 0.8750000 0.8333333 0.9166667 0.9285714 0.9142857 0.9428571 1584467623 0.9393083 0.9361198 0.9424969 0.9393686 0.9271963 0.9514204 0.9166667 0.9444444 0.8888889 0.9714286 1.0000000 0.9428571 1584468013 0.9999095 0.9998116 1.0000000 0.9993026 1.0000000 0.9986120 0.8888889 0.8055556 0.9722222 0.9571429 0.9142857 1.0000000
17 sigtopBonfSMOTE Yes Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p 1584463064 0.9956204 0.9951104 0.9961305 0.9787976 0.9814953 0.9761266 0.8888889 0.9166667 0.8611111 0.9714286 1.0000000 0.9428571 1584463520 0.9990513 0.9988740 0.9992287 0.9924211 1.0000000 0.9849172 0.8750000 0.8888889 0.8611111 0.9714286 1.0000000 0.9428571 1584464035 0.9985652 0.9982756 0.9988547 0.9917236 1.0000000 0.9835292 0.8888889 0.8888889 0.8888889 0.9714286 1.0000000 0.9428571 1584464606 1.0000000 1.0000000 1.0000000 0.9983261 1.0000000 0.9966688 0.9027778 0.8333333 0.9722222 0.9857143 0.9714286 1.0000000 1584465191 0.9948269 0.9941933 0.9954605 0.9789371 0.9814953 0.9764042 0.8888889 0.9166667 0.8611111 0.9714286 1.0000000 0.9428571 1584465666 1 1 1 1 1 1 0.8472222 0.7222222 0.9722222 0.9571429 0.9142857 1.0000000 1584466362 1.0000000 1.0000000 1.0000000 0.9999535 1.0000000 0.9999075 0.9166667 0.8611111 0.9722222 0.9857143 0.9714286 1.0000000 1584467254 0.9779993 0.9760129 0.9799858 0.9682429 0.9816822 0.9549366 0.8750000 0.8333333 0.9166667 0.9285714 0.9142857 0.9428571 1584467640 0.9393083 0.9361198 0.9424969 0.9393686 0.9271963 0.9514204 0.9166667 0.9444444 0.8888889 0.9714286 1.0000000 0.9428571 1584468032 0.9999095 0.9998116 1.0000000 0.9993026 1.0000000 0.9986120 0.8888889 0.8055556 0.9722222 0.9571429 0.9142857 1.0000000
18 sigtopHolmSMOTE Yes Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p 1584463084 0.9956204 0.9951104 0.9961305 0.9787976 0.9814953 0.9761266 0.8888889 0.9166667 0.8611111 0.9714286 1.0000000 0.9428571 1584463545 0.9978832 0.9975004 0.9982660 0.9884224 0.9908411 0.9860276 0.8750000 0.8888889 0.8611111 0.9571429 1.0000000 0.9142857 1584464061 0.9981969 0.9978385 0.9985553 0.9920491 1.0000000 0.9841769 0.8888889 0.8888889 0.8888889 0.9714286 1.0000000 0.9428571 1584464639 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.8750000 0.7777778 0.9722222 0.9714286 0.9428571 1.0000000 1584465217 0.9948269 0.9941933 0.9954605 0.9790301 0.9814953 0.9765892 0.8888889 0.9166667 0.8611111 0.9714286 1.0000000 0.9428571 1584465698 1 1 1 1 1 1 0.8611111 0.7500000 0.9722222 0.9571429 0.9142857 1.0000000 1584466430 1.0000000 1.0000000 1.0000000 0.9999535 1.0000000 0.9999075 0.9166667 0.8611111 0.9722222 0.9857143 0.9714286 1.0000000 1584467271 0.9779993 0.9760129 0.9799858 0.9682429 0.9816822 0.9549366 0.8750000 0.8333333 0.9166667 0.9285714 0.9142857 0.9428571 1584467656 0.9393083 0.9361198 0.9424969 0.9393686 0.9271963 0.9514204 0.9166667 0.9444444 0.8888889 0.9714286 1.0000000 0.9428571 1584468051 0.9999095 0.9998116 1.0000000 0.9993026 1.0000000 0.9986120 0.8888889 0.8055556 0.9722222 0.9571429 0.9142857 1.0000000
19 topFCSMOTE Yes Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p 1584463103 0.9978276 0.9973654 0.9982897 0.9933045 1.0000000 0.9866753 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584463571 0.9991518 0.9989223 0.9993814 0.9926536 0.9907477 0.9945406 0.9305556 0.9722222 0.8888889 1.0000000 1.0000000 1.0000000 1584464087 0.9991941 0.9989774 0.9994108 0.9969312 1.0000000 0.9938928 0.9305556 0.9722222 0.8888889 1.0000000 1.0000000 1.0000000 1584464669 0.9998316 0.9997065 0.9999567 0.9994420 1.0000000 0.9988896 0.9305556 0.9166667 0.9444444 1.0000000 1.0000000 1.0000000 1584465236 0.9976015 0.9971146 0.9980884 0.9935370 1.0000000 0.9871380 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584465728 1 1 1 1 1 1 0.9722222 1.0000000 0.9444444 0.9714286 0.9428571 1.0000000 1584466498 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9722222 1.0000000 0.9444444 0.9857143 1.0000000 0.9714286 1584467287 0.9920982 0.9909715 0.9932249 0.9913517 1.0000000 0.9827889 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584467673 0.9918549 0.9907060 0.9930039 0.9868880 0.9910280 0.9827889 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584468070 0.9999987 0.9999973 1.0000000 0.9998605 1.0000000 0.9997224 0.9166667 0.8888889 0.9444444 0.9571429 0.9142857 1.0000000
20 AUC_MDLSMOTE Yes Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.200a.5p + hsa.miR.127.5p 1584463123 0.9986041 0.9982479 0.9989603 0.9944669 1.0000000 0.9889886 0.9583333 1.0000000 0.9166667 0.9714286 1.0000000 0.9428571 1584463596 0.9962802 0.9954578 0.9971026 0.9935370 1.0000000 0.9871380 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464114 0.9979454 0.9972658 0.9986251 0.9954433 1.0000000 0.9909318 0.9444444 0.9722222 0.9166667 0.9714286 1.0000000 0.9428571 1584464699 0.9997539 0.9996008 0.9999069 0.9994420 1.0000000 0.9988896 0.9166667 0.8611111 0.9722222 0.9571429 0.9142857 1.0000000 1584465255 0.9981459 0.9976831 0.9986087 0.9939555 1.0000000 0.9879708 0.9583333 1.0000000 0.9166667 0.9714286 1.0000000 0.9428571 1584465759 1 1 1 1 1 1 0.9444444 0.9444444 0.9444444 0.9857143 0.9714286 1.0000000 1584466567 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9305556 0.9444444 0.9166667 1.0000000 1.0000000 1.0000000 1584467303 0.9774786 0.9755956 0.9793616 0.9755893 1.0000000 0.9514204 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584467689 0.9909485 0.9897382 0.9921589 0.9901892 1.0000000 0.9804756 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584468088 0.9999952 0.9999895 1.0000000 0.9998140 1.0000000 0.9996299 0.8888889 0.8333333 0.9444444 0.9285714 0.8857143 0.9714286
21 SU_MDLSMOTE Yes Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.708.3p + hsa.miR.375.3p + hsa.miR.30c.2.3p + hsa.miR.217.5p + hsa.miR.194.5p + hsa.miR.200a.3p + hsa.miR.132.3p + hsa.miR.199a.5p 1584463142 0.9989010 0.9985914 0.9992106 0.9940949 1.0000000 0.9882484 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584463622 0.9958725 0.9950065 0.9967385 0.9939090 1.0000000 0.9878782 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464140 0.9984466 0.9979116 0.9989817 0.9966523 1.0000000 0.9933377 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464730 0.9994697 0.9991769 0.9997624 0.9982796 1.0000000 0.9965763 0.9444444 0.9444444 0.9444444 1.0000000 1.0000000 1.0000000 1584465273 0.9984868 0.9980649 0.9989088 0.9947459 1.0000000 0.9895438 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584465791 1 1 1 1 1 1 0.9305556 0.9166667 0.9444444 0.9428571 0.8857143 1.0000000 1584466634 1.0000000 1.0000000 1.0000000 0.9999535 1.0000000 0.9999075 0.9444444 0.9722222 0.9166667 0.9571429 0.9428571 0.9714286 1584467320 0.9968461 0.9961296 0.9975626 0.9965128 1.0000000 0.9930601 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584467705 0.9909485 0.9897382 0.9921589 0.9901892 1.0000000 0.9804756 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584468107 0.9999760 0.9999595 0.9999924 0.9993955 1.0000000 0.9987971 0.8750000 0.8055556 0.9444444 0.9428571 0.9142857 0.9714286
22 CorrSF_MDLSMOTE Yes Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.708.3p + hsa.miR.30a.3p + hsa.miR.136.5p + hsa.miR.132.3p 1584463161 0.9969292 0.9962894 0.9975690 0.9845632 0.9913084 0.9778847 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584463646 0.9921348 0.9909225 0.9933470 0.9887014 0.9913084 0.9861201 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584464166 0.9964133 0.9957401 0.9970865 0.9889338 0.9913084 0.9865828 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584464763 0.9993308 0.9989900 0.9996717 0.9972102 1.0000000 0.9944480 0.9305556 0.9166667 0.9444444 0.9285714 0.8571429 1.0000000 1584465293 0.9964932 0.9958250 0.9971615 0.9863300 0.9913084 0.9814009 0.9444444 0.9722222 0.9166667 1.0000000 1.0000000 1.0000000 1584465814 1 1 1 1 1 1 0.9305556 0.8888889 0.9722222 0.9714286 0.9428571 1.0000000 1584466700 1.0000000 1.0000000 1.0000000 0.9996280 1.0000000 0.9992597 0.9305556 0.9166667 0.9444444 0.9857143 0.9714286 1.0000000 1584467336 0.9729217 0.9708267 0.9750166 0.9712652 0.9913084 0.9514204 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584467722 0.9866442 0.9851528 0.9881355 0.9858651 0.9913084 0.9804756 0.9305556 0.9444444 0.9166667 0.9857143 1.0000000 0.9714286 1584468126 0.9999149 0.9998332 0.9999967 0.9992096 1.0000000 0.9984269 0.9305556 0.9166667 0.9444444 0.9428571 0.9428571 0.9428571
23 AUC_MDL_sig No Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p 1584463178 0.9931872 0.9810931 1.0000000 0.9906542 1.0000000 0.9813084 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584463659 0.9902612 0.9763623 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464178 0.9896934 0.9746777 1.0000000 0.9813084 0.9813084 0.9813084 0.9722222 1.0000000 0.9444444 0.9857143 1.0000000 0.9714286 1584464781 0.9910036 0.9740380 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584465307 0.9894314 0.9729248 1.0000000 0.9906542 1.0000000 0.9813084 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584465826 1 1 1 1 1 1 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584466713 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584467350 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467736 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584468143 0.9941480 0.9887442 0.9995518 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
24 SU_MDL_sig No Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.30a.3p + hsa.miR.10b.5p + hsa.miR.126.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.126.3p 1584463185 0.9965936 0.9915862 1.0000000 0.9766355 0.9813084 0.9719626 0.9305556 0.9444444 0.9166667 0.9714286 1.0000000 0.9428571 1584463666 0.9928378 0.9815851 1.0000000 0.9813084 0.9906542 0.9719626 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584464186 0.9925758 0.9812489 1.0000000 0.9392523 1.0000000 0.8785047 0.8888889 1.0000000 0.7777778 0.9142857 1.0000000 0.8285714 1584464789 0.9975544 0.9928587 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584465315 0.9937986 0.9837316 1.0000000 0.9813084 0.9906542 0.9719626 0.9444444 0.9722222 0.9166667 0.9714286 1.0000000 0.9428571 1584465834 1 1 1 1 1 1 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584466721 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9722222 1.0000000 0.9444444 0.9571429 0.9714286 0.9428571 1584467358 0.9889073 0.9755302 1.0000000 0.9672897 0.9439252 0.9906542 0.9305556 0.9444444 0.9166667 0.9714286 0.9714286 0.9714286 1584467744 0.9889073 0.9755302 1.0000000 0.9672897 0.9439252 0.9906542 0.9305556 0.9444444 0.9166667 0.9714286 0.9714286 0.9714286 1584468151 0.9825749 0.9652791 0.9998707 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
25 CorrSF_MDL_sig No Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.194.5p + hsa.miR.10b.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.338.3p + hsa.miR.217.5p + hsa.miR.378a.5p + hsa.miR.195.5p 1584463193 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584463674 0.9969430 0.9916498 1.0000000 0.9766355 0.9719626 0.9813084 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584464194 0.9962442 0.9898840 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464797 1.0000000 1.0000000 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584465323 0.9976417 0.9931160 1.0000000 0.9906542 1.0000000 0.9813084 0.9305556 0.9722222 0.8888889 0.9857143 1.0000000 0.9714286 1584465842 1 1 1 1 1 1 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584466729 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9722222 1.0000000 0.9444444 0.9571429 0.9714286 0.9428571 1584467366 0.9889073 0.9755302 1.0000000 0.9672897 0.9439252 0.9906542 0.9166667 0.9166667 0.9166667 0.9571429 0.9714286 0.9428571 1584467752 0.9889073 0.9755302 1.0000000 0.9672897 0.9439252 0.9906542 0.9166667 0.9166667 0.9166667 0.9571429 0.9714286 0.9428571 1584468158 0.9942790 0.9890146 0.9995433 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
26 AUC_MDLSMOTE_sig Yes Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.127.5p + hsa.miR.139.5p 1584463202 0.9981693 0.9976671 0.9986715 0.9943739 1.0000000 0.9888036 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584463695 0.9948446 0.9938796 0.9958095 0.9927930 1.0000000 0.9856574 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464216 0.9997759 0.9996748 0.9998771 0.9961408 1.0000000 0.9923198 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584464818 1.0000000 1.0000000 1.0000000 0.9999535 1.0000000 0.9999075 0.9305556 0.8888889 0.9722222 0.9571429 0.9142857 1.0000000 1584465334 0.9975346 0.9968667 0.9982025 0.9940484 1.0000000 0.9881558 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584465869 1 1 1 1 1 1 0.9583333 0.9444444 0.9722222 1.0000000 1.0000000 1.0000000 1584466792 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 0.9722222 0.9166667 1.0000000 1.0000000 1.0000000 1584467375 0.9774786 0.9755956 0.9793616 0.9755893 1.0000000 0.9514204 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584467761 0.9909485 0.9897382 0.9921589 0.9901892 1.0000000 0.9804756 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584468167 0.9999504 0.9998922 1.0000000 0.9993490 1.0000000 0.9987045 0.9027778 0.8611111 0.9444444 0.9428571 0.9142857 0.9714286
27 SU_MDLSMOTE_sig Yes Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.375.3p + hsa.miR.708.3p + hsa.miR.136.5p + hsa.miR.30c.2.3p + hsa.miR.194.5p + hsa.miR.200a.3p + hsa.miR.217.5p + hsa.miR.132.3p 1584463221 0.9991771 0.9989537 0.9994004 0.9942344 1.0000000 0.9885260 0.9583333 0.9722222 0.9444444 0.9857143 1.0000000 0.9714286 1584463720 0.9947933 0.9938227 0.9957639 0.9901892 1.0000000 0.9804756 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584464241 0.9999494 0.9999143 0.9999845 0.9993490 1.0000000 0.9987045 0.9444444 0.9722222 0.9166667 1.0000000 1.0000000 1.0000000 1584464850 0.9998279 0.9996898 0.9999661 0.9993026 1.0000000 0.9986120 0.9583333 0.9444444 0.9722222 0.9714286 0.9428571 1.0000000 1584465353 0.9989748 0.9986937 0.9992560 0.9946994 1.0000000 0.9894513 0.9583333 0.9722222 0.9444444 0.9857143 1.0000000 0.9714286 1584465899 1 1 1 1 1 1 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584466860 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9305556 0.9444444 0.9166667 0.9857143 1.0000000 0.9714286 1584467392 0.9968461 0.9961296 0.9975626 0.9965128 1.0000000 0.9930601 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584467778 0.9909485 0.9897382 0.9921589 0.9901892 1.0000000 0.9804756 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584468186 0.9999760 0.9999595 0.9999924 0.9993955 1.0000000 0.9987971 0.8750000 0.8055556 0.9444444 0.9428571 0.9142857 0.9714286
28 CorrSF_MDLSMOTE_sig Yes Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30a.3p 1584463240 0.9914422 0.9902532 0.9926312 0.9824708 0.9913084 0.9737207 0.9583333 1.0000000 0.9166667 0.9714286 1.0000000 0.9428571 1584463745 0.9917561 0.9906045 0.9929077 0.9810759 0.9913084 0.9709448 0.9583333 1.0000000 0.9166667 0.9714286 1.0000000 0.9428571 1584464266 0.9918466 0.9907094 0.9929837 0.9732180 0.9652336 0.9811233 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584464888 0.9968592 0.9961564 0.9975620 0.9924211 1.0000000 0.9849172 0.9305556 0.9166667 0.9444444 0.9571429 0.9428571 0.9714286 1584465373 0.9906387 0.9893655 0.9919119 0.9823778 0.9913084 0.9735357 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584465919 1 1 1 1 1 1 0.9305556 0.8888889 0.9722222 0.9142857 0.8571429 0.9714286 1584466923 0.9998515 0.9997596 0.9999434 0.9989771 1.0000000 0.9979643 0.9166667 0.8888889 0.9444444 0.9428571 0.9142857 0.9714286 1584467408 0.9729217 0.9708267 0.9750166 0.9712652 0.9913084 0.9514204 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584467794 0.9729217 0.9708267 0.9750166 0.9712652 0.9913084 0.9514204 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584468205 0.9996431 0.9995127 0.9997735 0.9974892 1.0000000 0.9950032 0.9444444 0.9166667 0.9722222 0.9428571 0.9428571 0.9428571
29 RandomForestRFE No Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.375.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.217.5p + hsa.miR.194.5p + hsa.miR.192.5p + hsa.miR.126.5p 1584463255 0.9963316 0.9902920 1.0000000 0.9859813 1.0000000 0.9719626 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584463757 0.9917897 0.9780033 1.0000000 0.9813084 0.9813084 0.9813084 0.9722222 1.0000000 0.9444444 0.9857143 1.0000000 0.9714286 1584464279 0.9910909 0.9766273 1.0000000 0.9719626 0.9626168 0.9813084 0.9722222 1.0000000 0.9444444 0.9857143 1.0000000 0.9714286 1584464905 0.9955455 0.9867696 1.0000000 0.9906542 1.0000000 0.9813084 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584465387 0.9923137 0.9797836 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584465931 1 1 1 1 1 1 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584466934 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584467421 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467807 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584468221 0.9941480 0.9887442 0.9995518 0.9579439 0.9345794 0.9813084 0.9166667 0.9444444 0.8888889 0.9571429 0.9714286 0.9428571
30 cfsSMOTE Yes Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.132.3p + hsa.miR.136.5p + hsa.miR.30e.3p 1584463264 0.9965618 0.9958880 0.9972356 0.9843307 0.9913084 0.9774220 0.9444444 0.9722222 0.9166667 0.9714286 1.0000000 0.9428571 1584463777 0.9956546 0.9948894 0.9964198 0.9888408 0.9913084 0.9863977 0.9444444 0.9444444 0.9444444 0.9857143 1.0000000 0.9714286 1584464299 0.9913708 0.9900964 0.9926452 0.9886084 0.9913084 0.9859350 0.9722222 1.0000000 0.9444444 0.9714286 1.0000000 0.9428571 1584464929 0.9982670 0.9977489 0.9987851 0.9942344 1.0000000 0.9885260 0.9305556 0.9166667 0.9444444 0.9571429 0.9428571 0.9714286 1584465400 0.9957081 0.9949285 0.9964877 0.9865625 0.9913084 0.9818636 0.9583333 1.0000000 0.9166667 0.9571429 1.0000000 0.9142857 1584465950 1 1 1 1 1 1 0.9166667 0.8888889 0.9444444 0.9428571 0.8857143 1.0000000 1584466995 1.0000000 1.0000000 1.0000000 0.9997675 1.0000000 0.9995373 0.9305556 0.9166667 0.9444444 0.9571429 0.9428571 0.9714286 1584467430 0.9729217 0.9708267 0.9750166 0.9712652 0.9913084 0.9514204 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584467816 0.9866442 0.9851528 0.9881355 0.9858651 0.9913084 0.9804756 0.9305556 0.9444444 0.9166667 0.9857143 1.0000000 0.9714286 1584468230 0.9997259 0.9995436 0.9999082 0.9989306 1.0000000 0.9978717 0.9444444 0.9444444 0.9444444 0.9000000 0.8571429 0.9428571
31 cfsSMOTE_sig Yes Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.132.3p + hsa.miR.136.5p 1584463281 0.9965618 0.9958880 0.9972356 0.9843307 0.9913084 0.9774220 0.9444444 0.9722222 0.9166667 0.9714286 1.0000000 0.9428571 1584463801 0.9922008 0.9909959 0.9934058 0.9764728 0.9639252 0.9888961 0.9444444 0.9444444 0.9444444 1.0000000 1.0000000 1.0000000 1584464322 0.9909486 0.9896368 0.9922603 0.9870740 0.9913084 0.9828815 0.9722222 1.0000000 0.9444444 0.9714286 1.0000000 0.9428571 1584464961 0.9982681 0.9977503 0.9987860 0.9941879 1.0000000 0.9884334 0.9305556 0.9166667 0.9444444 0.9571429 0.9428571 0.9714286 1584465419 0.9957081 0.9949285 0.9964877 0.9866090 0.9913084 0.9819561 0.9583333 1.0000000 0.9166667 0.9571429 1.0000000 0.9142857 1584465973 1 1 1 1 1 1 0.9166667 0.8888889 0.9444444 0.9428571 0.8857143 1.0000000 1584467060 1.0000000 1.0000000 1.0000000 0.9997675 1.0000000 0.9995373 0.9305556 0.9166667 0.9444444 0.9571429 0.9428571 0.9714286 1584467445 0.9729217 0.9708267 0.9750166 0.9712652 0.9913084 0.9514204 0.9305556 0.9722222 0.8888889 0.9714286 1.0000000 0.9428571 1584467831 0.9866442 0.9851528 0.9881355 0.9858651 0.9913084 0.9804756 0.9305556 0.9444444 0.9166667 0.9857143 1.0000000 0.9714286 1584468247 0.9997259 0.9995436 0.9999082 0.9989306 1.0000000 0.9978717 0.9444444 0.9444444 0.9444444 0.9000000 0.8571429 0.9428571
32 Mystepwise_glm_binomial No Class ~ hsa.miR.21.5p + hsa.miR.130a.3p + hsa.miR.30e.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.146b.5p 1584463298 0.9995633 0.9987652 1.0000000 0.9906542 0.9906542 0.9906542 0.9583333 0.9722222 0.9444444 1.0000000 1.0000000 1.0000000 1584463814 0.9971177 0.9924022 1.0000000 0.9906542 1.0000000 0.9813084 0.9444444 1.0000000 0.8888889 0.9857143 1.0000000 0.9714286 1584464335 0.9976417 0.9938877 1.0000000 0.9906542 1.0000000 0.9813084 0.9583333 1.0000000 0.9166667 0.9857143 1.0000000 0.9714286 1584464978 0.9994759 0.9983702 1.0000000 0.9906542 0.9906542 0.9906542 0.9444444 0.9722222 0.9166667 1.0000000 1.0000000 1.0000000 1584465434 0.9972050 0.9925361 1.0000000 0.9859813 0.9906542 0.9813084 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584465985 1 1 1 1 1 1 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584467072 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584467458 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584467845 0.9939296 0.9848574 1.0000000 0.9719626 0.9532710 0.9906542 0.9444444 0.9722222 0.9166667 0.9714286 0.9714286 0.9714286 1584468263 0.9945410 0.9895576 0.9995244 0.9579439 0.9345794 0.9813084 0.9166667 0.9166667 0.9166667 0.9428571 0.9714286 0.9142857
33 Mystepwise_sig_glm_binomial No Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p 1584463306 0.9999127 0.9996706 1.0000000 0.9906542 0.9906542 0.9906542 0.9583333 1.0000000 0.9166667 1.0000000 1.0000000 1.0000000 1584463821 0.9998253 0.9994066 1.0000000 0.9906542 0.9813084 1.0000000 0.9444444 0.9722222 0.9166667 1.0000000 1.0000000 1.0000000 1584464343 0.9997380 0.9991987 1.0000000 0.9906542 0.9906542 0.9906542 0.9444444 0.9722222 0.9166667 1.0000000 1.0000000 1.0000000 1584464987 0.9998253 0.9994066 1.0000000 0.9953271 1.0000000 0.9906542 0.9722222 1.0000000 0.9444444 1.0000000 1.0000000 1.0000000 1584465442 0.9997380 0.9991987 1.0000000 0.9906542 0.9906542 0.9906542 0.9444444 0.9722222 0.9166667 0.9857143 1.0000000 0.9714286 1584465993 1 1 1 1 1 1 0.9722222 1.0000000 0.9444444 0.9857143 1.0000000 0.9714286 1584467080 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000 0.9722222 1.0000000 0.9444444 0.9857143 1.0000000 0.9714286 1584467466 0.9816578 0.9643889 0.9989267 0.9672897 0.9719626 0.9626168 0.9861111 1.0000000 0.9722222 0.9857143 1.0000000 0.9714286 1584467853 0.9654118 0.9421733 0.9886504 0.9626168 1.0000000 0.9252336 0.9444444 1.0000000 0.8888889 0.9714286 1.0000000 0.9428571 1584468270 0.9888200 0.9767842 1.0000000 0.9532710 0.9532710 0.9532710 0.9583333 1.0000000 0.9166667 0.9428571 1.0000000 0.8857143

Description of columns:

By this logic every parameter is also calculated from testing (_test_) and validation (_valid_) set. If the method generated a probability, a default cutoff is used for all of the predictions.

Let’s see the general performance (accuracy) of methods in the benchmark:

metody = ks.get_benchmark_methods("benchmark.csv")  # gets the methods used in benchmark
par(mfrow = c(2, 2))
for (i in 1:length(metody)) {
    temp = ks.get_benchmark("benchmark.csv")  # loads benchmark
    temp2 = dplyr::select(temp, starts_with(paste0(metody[i], "_")))
    boxplot(temp[, paste0(metody[i], "_train_Accuracy")], temp[, paste0(metody[i], "_test_Accuracy")], temp[, paste0(metody[i], "_valid_Accuracy")], main = metody[i], 
        names = c("Training", "Testing", "Validation"), ylab = "Accuracy", ylim = c(0.5, 1))
    tempids = c(match(paste0(metody[i], "_train_Accuracy"), colnames(temp)), match(paste0(metody[i], "_test_Accuracy"), colnames(temp)), match(paste0(metody[i], 
        "_valid_Accuracy"), colnames(temp)))
}

par(mfrow = c(1, 1))

In this package, the best signature can be selected using 3 methods:

1. The signture which achived the best accuracy in training, testing and validation: (metaindex = mean of all 3 accuracy metrics)

acc1 = ks.best_signiture_proposals(benchmark_csv = "benchmark.csv", without_train = F)  # generates the benchmark sorted by metaindex
best_signatures = acc1[1:3, ]  # get top 3 methods
knitr::kable(best_signatures[, 31:33])
metaindex method miRy
0.9775265 topFCSMOTE Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p
0.9765162 Mystepwise_sig_glm_binomial Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p
0.9741000 SU_MDLSMOTE_sig Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.375.3p + hsa.miR.708.3p + hsa.miR.136.5p + hsa.miR.30c.2.3p + hsa.miR.194.5p + hsa.miR.200a.3p + hsa.miR.217.5p + hsa.miR.132.3p

2. The signture which achived the best accuracy in testing and validation: (metaindex = mean of 2 accuracy metrics)

acc1 = ks.best_signiture_proposals(benchmark_csv = "benchmark.csv", without_train = T)  # generates the benchmark sorted by metaindex
best_signatures = acc1[1:3, ]  # get top 3 methods
knitr::kable(best_signatures[, 21:23])
metaindex method miRy
0.9727183 Mystepwise_sig_glm_binomial Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p
0.9685913 topFCSMOTE Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p
0.9671429 AUC_MDL Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p

3. The signture which achived the best sensitivity and specificity in validation: (metaindex = mean of sensivitiy and specificity in validation dataset)

acc = ks.best_signiture_proposals_meta11(benchmark_csv = "benchmark.csv")  # generates the benchmark sorted by metaindex
best_signatures = acc[1:3, ]  # get top 3 methods
knitr::kable(best_signatures[, c(2:4, 135)])
method SMOTE miRy metaindex
topFCSMOTE Yes Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p 0.9771429
Mystepwise_sig_glm_binomial No Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p 0.9714286
AUC_MDL No Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p 0.9685714

Let’s assess the over/underfitting of selected methods for top 2 signatures. The plots show the change of accuracy between datasets across top 5 methods of feature selection.

for (i in 1:length(metody)) {
    suppressMessages(library(PairedData))
    suppressMessages(library(profileR))
    pd = paired(as.numeric(acc[1:5, paste0(metody[i], "_train_Accuracy")]), as.numeric(acc[1:5, paste0(metody[i], "_test_Accuracy")]))
    colnames(pd) = c("Train Acc", "Test Acc")
    plot2 = profileplot(pd, person.id = acc$method[1:5], standardize = F)
    pd = paired(as.numeric(acc[1:5, paste0(metody[i], "_train_Accuracy")]), as.numeric(acc[1:5, paste0(metody[i], "_valid_Accuracy")]))
    colnames(pd) = c("Train Acc", "Valid Acc")
    plot3 = profileplot(pd, person.id = acc$method[1:5], standardize = F)
    pd = paired(as.numeric(acc[1:5, paste0(metody[i], "_test_Accuracy")]), as.numeric(acc[1:5, paste0(metody[i], "_valid_Accuracy")]))
    colnames(pd) = c("Test Acc", "Valid Acc")
    plot4 = profileplot(pd, person.id = acc$method[1:5], standardize = F)
    
    
    
    require(gridExtra)
    grid.arrange(arrangeGrob(plot2, plot3, ncol = 2, nrow = 1, top = metody[i]))
    grid.arrange(arrangeGrob(plot4, ncol = 1, nrow = 1, top = metody[i]))
}

The relationship betweend accuracy on testin and training set can be further visualized as follows:

acc2 = acc[1:6, ]  # get top 6 methods
accmelt = melt(acc2, id.vars = "method") %>% filter(variable != "metaindex") %>% filter(variable != "miRy")
accmelt = cbind(accmelt, strsplit2(accmelt$variable, "_"))
acctest = accmelt$value[accmelt$`2` == "test"]
accvalid = accmelt$value[accmelt$`2` == "valid"]
accmeth = accmelt$method[accmelt$`2` == "test"]
unique(accmeth)
#> [1] "topFCSMOTE"                  "Mystepwise_sig_glm_binomial" "AUC_MDL"                     "SU_MDLSMOTE_sig"             "cfs_sig"                    
#> [6] "AUC_MDL_sig"
plot5 = ggplot(, aes(x = as.numeric(acctest), y = as.numeric(accvalid), shape = accmeth)) + geom_point() + scale_x_continuous(name = "Accuracy on test set", 
    limits = c(0.5, 1)) + scale_y_continuous(name = "Accuracy on validation set", limits = c(0.5, 1)) + theme_bw()
grid.arrange(arrangeGrob(plot5, ncol = 1, nrow = 1))

Best signture analysis

Suppose we chose to select the best signitures based on the best sensitivity and specificity in validation. Let’s see 3 best signutures:

kable(best_signatures[1:3, 2:4])
method SMOTE miRy
topFCSMOTE Yes Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p
Mystepwise_sig_glm_binomial No Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p
AUC_MDL No Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p

To get the miRNAs from formula you can use ks.get_miRNAs_from_benchmark.

selected_miRNAs = ks.get_miRNAs_from_benchmark(benchmark_csv = "benchmark.csv", best_signatures$method[1])  # for the best performing signiture
selected_miRNAs
#>  [1] "hsa.miR.217.5p"   "hsa.miR.375.3p"   "hsa.miR.192.5p"   "hsa.miR.194.5p"   "hsa.miR.21.5p"    "hsa.miR.30a.3p"   "hsa.miR.199b.5p"  "hsa.miR.144.5p"   "hsa.miR.30c.2.3p"
#> [10] "hsa.miR.200a.3p"  "hsa.miR.708.3p"

Let’s check the differential expression metric of selected miRNAs:

best_de = ks.best_signiture_de(selected_miRNAs)
ks.table(best_de)
miR log2FC p-value BH
7 hsa.miR.217.5p 4.996853 0
6 hsa.miR.375.3p 4.788105 0
23 hsa.miR.192.5p 2.963697 0
18 hsa.miR.194.5p 2.943915 0
1 hsa.miR.21.5p 2.602216 0
3 hsa.miR.30a.3p -2.441965 0
11 hsa.miR.199b.5p 2.381457 0
39 hsa.miR.144.5p -2.112787 0
4 hsa.miR.30c.2.3p -2.082431 0
25 hsa.miR.200a.3p 2.194942 0
30 hsa.miR.708.3p 2.162216 0

Let’s visualize the performance of those methods using barplots:

for (i in 1:3) {
    cat(paste0("\n\n## ", acc$method[i], "\n\n"))
    par(mfrow = c(1, 2))
    acc = ks.best_signiture_proposals_meta11("benchmark.csv")
    metody = ks.get_benchmark_methods("benchmark.csv")
    ktory_set = match(acc$method[i], ks.get_benchmark("benchmark.csv")$method)
    # do_ktorej_kolumny = which(colnames(acc) == 'metaindex') barplot(as.numeric(acc[i,1:do_ktorej_kolumny]))
    for (ii in 1:length(metody)) {
        
        temp = ks.get_benchmark("benchmark.csv") %>% dplyr::select(starts_with(paste0(metody[ii], "_t")), starts_with(paste0(metody[ii], "_v")))
        
        ROCtext = paste0("Training AUC ROC: ", round(temp[ktory_set, 1], 2), " (95%CI: ", round(temp[ktory_set, 2], 2), "-", round(temp[ktory_set, 3], 
            2), ")")
        
        temp = temp[, -c(1:3)]
        temp2 = as.numeric(temp[ktory_set, ])
        temp3 = matrix(temp2, nrow = 3, byrow = T)
        colnames(temp3) = c("Accuracy", "Sensitivity", "Specificity")
        rownames(temp3) = c("Training", "Testing", "Validation")
        temp3 = t(temp3)
        
        plot1 = barplot(temp3, beside = T, ylim = c(0, 1), xlab = paste0(ROCtext, "\nBlack - accuracy, blue - sensitivity, green - specificity"), width = 0.85, 
            col = c("black", "blue", "green"), legend = F, args.legend = list(x = "topright", bty = "n", inset = c(0, -0.25)), cex.lab = 0.7, main = paste0(acc$method[i], 
                " - ", metody[ii]), font.lab = 2)
        ## Add text at top of bars
        text(x = plot1, y = as.numeric(temp3), label = paste0(round(as.numeric(temp[ktory_set, ]) * 100, 1), "%"), pos = 3, cex = 0.6, col = "red")
    }
    par(mfrow = c(1, 1))
    
}
#> 
#> 
#> ## topFCSMOTE

#> 
#> 
#> ## Mystepwise_sig_glm_binomial

#> 
#> 
#> ## AUC_MDL

Finally, we can assess the overlap of top 3 feature selection methods:

overlap = ks.miRNA_signiture_overlap(acc$method[1:3], "benchmark.csv")

Which 3 miRNAs are common for all 3 signatures?

attr(overlap, "intersections")$`topFCSMOTE:Mystepwise_sig_glm_binomial:AUC_MDL`
#> [1] "hsa.miR.375.3p"   "hsa.miR.21.5p"    "hsa.miR.30c.2.3p"

Let’s draw vulcano plot and mark the miRNAs selected in best signature:

ks.vulcano_plot(selected_miRNAs = de$miR, DE = de, only_label = selected_miRNAs)

Let’s draw heatmap for selected miRNAs in whole dataset (training, testing and validation set).

ks.heatmap(x = dplyr::select(mixed, selected_miRNAs), rlab = data.frame(Class = mixed$Class, Mix = mixed$mix), zscore = F, margins = c(10, 10))

ks.heatmap(x = dplyr::select(mixed, selected_miRNAs), rlab = data.frame(Class = mixed$Class, Mix = mixed$mix), zscore = T, margins = c(10, 10))

Based on everything we have done so far, we suggest using the following signiture in further validation of biomarker study.

cat(paste0(selected_miRNAs, collapse = ", "))
#> hsa.miR.217.5p, hsa.miR.375.3p, hsa.miR.192.5p, hsa.miR.194.5p, hsa.miR.21.5p, hsa.miR.30a.3p, hsa.miR.199b.5p, hsa.miR.144.5p, hsa.miR.30c.2.3p, hsa.miR.200a.3p, hsa.miR.708.3p

Sesssion

session_info()
#> ─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.2 (2019-12-12)
#>  os       Ubuntu 18.04.4 LTS          
#>  system   x86_64, linux-gnu           
#>  ui       RStudio                     
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       Europe/Warsaw               
#>  date     2020-03-18                  
#> 
#> ─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  package          * version    date       lib source                           
#>  abind              1.4-5      2016-07-21 [1] CRAN (R 3.6.0)                   
#>  arules             1.6-4      2019-08-29 [1] CRAN (R 3.6.1)                   
#>  assertthat         0.2.1      2019-03-21 [1] CRAN (R 3.6.0)                   
#>  backports          1.1.5      2019-10-02 [1] CRAN (R 3.6.1)                   
#>  Biocomb          * 0.4        2018-05-18 [1] CRAN (R 3.6.2)                   
#>  bitops             1.0-6      2013-08-17 [1] CRAN (R 3.6.0)                   
#>  boot               1.3-24     2019-12-20 [4] CRAN (R 3.6.2)                   
#>  bounceR            0.1        2019-07-29 [1] Github (STATWORX/bounceR@0f714f3)
#>  broom              0.5.5      2020-02-29 [1] CRAN (R 3.6.2)                   
#>  callr              3.4.2      2020-02-12 [1] CRAN (R 3.6.2)                   
#>  caret            * 6.0-85     2020-01-07 [1] CRAN (R 3.6.2)                   
#>  caTools            1.18.0     2020-01-17 [1] CRAN (R 3.6.1)                   
#>  cellranger         1.1.0      2016-07-27 [1] CRAN (R 3.6.0)                   
#>  class              7.3-15     2019-01-01 [4] CRAN (R 3.6.0)                   
#>  classInt         * 0.4-2      2019-10-17 [1] CRAN (R 3.6.2)                   
#>  cli                2.0.2      2020-02-28 [1] CRAN (R 3.6.2)                   
#>  cluster            2.1.0      2019-06-19 [1] CRAN (R 3.6.1)                   
#>  codetools          0.2-16     2018-12-24 [4] CRAN (R 3.6.0)                   
#>  colorspace         1.4-1      2019-03-18 [1] CRAN (R 3.6.0)                   
#>  crayon             1.3.4      2017-09-16 [1] CRAN (R 3.6.0)                   
#>  crosstalk          1.1.0.1    2020-03-13 [1] CRAN (R 3.6.2)                   
#>  cubature           2.0.4      2019-12-04 [1] CRAN (R 3.6.1)                   
#>  curl               4.3        2019-12-02 [1] CRAN (R 3.6.1)                   
#>  data.table       * 1.12.8     2019-12-09 [1] CRAN (R 3.6.2)                   
#>  DBI                1.1.0      2019-12-15 [1] CRAN (R 3.6.1)                   
#>  dbplyr             1.4.2      2019-06-17 [1] CRAN (R 3.6.1)                   
#>  desc               1.2.0      2018-05-01 [1] CRAN (R 3.6.0)                   
#>  devtools         * 2.2.2      2020-02-17 [1] CRAN (R 3.6.2)                   
#>  digest             0.6.25     2020-02-23 [1] CRAN (R 3.6.2)                   
#>  DMwR             * 0.4.1      2013-08-08 [1] CRAN (R 3.6.2)                   
#>  doParallel       * 1.0.15     2019-08-02 [1] CRAN (R 3.6.2)                   
#>  dplyr            * 0.8.5      2020-03-07 [1] CRAN (R 3.6.2)                   
#>  e1071              1.7-3      2019-11-26 [1] CRAN (R 3.6.1)                   
#>  edgeR            * 3.26.8     2019-09-01 [1] Bioconductor                     
#>  ellipsis           0.3.0      2019-09-20 [1] CRAN (R 3.6.1)                   
#>  entropy            1.2.1      2014-11-14 [1] CRAN (R 3.6.1)                   
#>  epiDisplay       * 3.5.0.1    2018-05-10 [1] CRAN (R 3.6.2)                   
#>  evaluate           0.14       2019-05-28 [1] CRAN (R 3.6.0)                   
#>  fansi              0.4.1      2020-01-08 [1] CRAN (R 3.6.1)                   
#>  farver             2.0.3      2020-01-16 [1] CRAN (R 3.6.1)                   
#>  fastmap            1.0.1      2019-10-08 [1] CRAN (R 3.6.1)                   
#>  forcats          * 0.5.0      2020-03-01 [1] CRAN (R 3.6.2)                   
#>  foreach          * 1.4.8      2020-02-09 [1] CRAN (R 3.6.2)                   
#>  foreign          * 0.8-76     2020-03-03 [4] CRAN (R 3.6.2)                   
#>  formatR            1.7        2019-06-11 [1] CRAN (R 3.6.0)                   
#>  Formula            1.2-3      2018-05-03 [1] CRAN (R 3.6.0)                   
#>  fs                 1.3.2      2020-03-05 [1] CRAN (R 3.6.2)                   
#>  FSelector          0.31       2018-05-16 [1] CRAN (R 3.6.1)                   
#>  furrr              0.1.0      2018-05-16 [1] CRAN (R 3.6.1)                   
#>  future             1.16.0     2020-01-16 [1] CRAN (R 3.6.1)                   
#>  gdata              2.18.0     2017-06-06 [1] CRAN (R 3.6.0)                   
#>  generics           0.0.2      2018-11-29 [1] CRAN (R 3.6.0)                   
#>  ggbiplot         * 0.55       2019-08-26 [1] Github (vqv/ggbiplot@7325e88)    
#>  ggplot2          * 3.3.0      2020-03-05 [1] CRAN (R 3.6.2)                   
#>  ggrepel          * 0.8.2      2020-03-08 [1] CRAN (R 3.6.2)                   
#>  gld              * 2.6.2      2020-01-08 [1] CRAN (R 3.6.1)                   
#>  globals            0.12.5     2019-12-07 [1] CRAN (R 3.6.1)                   
#>  glue               1.3.2      2020-03-12 [1] CRAN (R 3.6.2)                   
#>  gower              0.2.1      2019-05-14 [1] CRAN (R 3.6.0)                   
#>  gplots           * 3.0.3      2020-02-25 [1] CRAN (R 3.6.2)                   
#>  gridExtra        * 2.3        2017-09-09 [1] CRAN (R 3.6.2)                   
#>  gtable             0.3.0      2019-03-25 [1] CRAN (R 3.6.0)                   
#>  gtools           * 3.8.1      2018-06-26 [1] CRAN (R 3.6.0)                   
#>  haven              2.2.0      2019-11-08 [1] CRAN (R 3.6.1)                   
#>  highr              0.8        2019-03-20 [1] CRAN (R 3.6.0)                   
#>  hms                0.5.3      2020-01-08 [1] CRAN (R 3.6.1)                   
#>  htmltools          0.4.0      2019-10-04 [1] CRAN (R 3.6.1)                   
#>  htmlwidgets        1.5.1      2019-10-08 [1] CRAN (R 3.6.1)                   
#>  httpuv             1.5.2      2019-09-11 [1] CRAN (R 3.6.1)                   
#>  httr               1.4.1      2019-08-05 [1] CRAN (R 3.6.1)                   
#>  igraph             1.2.4.2    2019-11-27 [1] CRAN (R 3.6.1)                   
#>  inum               1.0-1      2019-04-25 [1] CRAN (R 3.6.0)                   
#>  ipred              0.9-9      2019-04-28 [1] CRAN (R 3.6.0)                   
#>  iterators        * 1.0.12     2019-07-26 [1] CRAN (R 3.6.1)                   
#>  jsonlite           1.6.1      2020-02-02 [1] CRAN (R 3.6.2)                   
#>  kableExtra       * 1.1.0      2019-03-16 [1] CRAN (R 3.6.0)                   
#>  KernSmooth         2.23-16    2019-10-15 [4] CRAN (R 3.6.1)                   
#>  knitr            * 1.28       2020-02-06 [1] CRAN (R 3.6.2)                   
#>  labeling           0.3        2014-08-23 [1] CRAN (R 3.6.0)                   
#>  later              1.0.0      2019-10-04 [1] CRAN (R 3.6.1)                   
#>  lattice          * 0.20-40    2020-02-19 [4] CRAN (R 3.6.2)                   
#>  lava               1.6.7      2020-03-05 [1] CRAN (R 3.6.2)                   
#>  lavaan           * 0.6-5      2019-08-28 [1] CRAN (R 3.6.1)                   
#>  lazyeval           0.2.2      2019-03-15 [1] CRAN (R 3.6.0)                   
#>  libcoin            1.0-5      2019-08-27 [1] CRAN (R 3.6.1)                   
#>  lifecycle          0.2.0      2020-03-06 [1] CRAN (R 3.6.2)                   
#>  limma            * 3.40.6     2019-07-26 [1] Bioconductor                     
#>  listenv            0.8.0      2019-12-05 [1] CRAN (R 3.6.1)                   
#>  lmom               2.8        2019-03-12 [1] CRAN (R 3.6.1)                   
#>  locfit             1.5-9.1    2013-04-20 [1] CRAN (R 3.6.0)                   
#>  lubridate          1.7.4      2018-04-11 [1] CRAN (R 3.6.0)                   
#>  magrittr           1.5        2014-11-22 [1] CRAN (R 3.6.0)                   
#>  manipulateWidget   0.10.1     2020-02-24 [1] CRAN (R 3.6.2)                   
#>  MASS             * 7.3-51.5   2019-12-20 [1] CRAN (R 3.6.2)                   
#>  MatchIt          * 3.0.2      2018-01-09 [1] CRAN (R 3.6.2)                   
#>  Matrix             1.2-18     2019-11-27 [4] CRAN (R 3.6.1)                   
#>  MatrixModels       0.4-1      2015-08-22 [1] CRAN (R 3.6.0)                   
#>  mboost             2.9-2      2020-02-18 [1] CRAN (R 3.6.2)                   
#>  memoise            1.1.0      2017-04-21 [1] CRAN (R 3.6.0)                   
#>  mice             * 3.8.0      2020-02-21 [1] CRAN (R 3.6.2)                   
#>  mime               0.9        2020-02-04 [1] CRAN (R 3.6.2)                   
#>  miniUI             0.1.1.1    2018-05-18 [1] CRAN (R 3.6.0)                   
#>  miRNAselector    * 0.1.0      2020-03-18 [1] local                            
#>  mnormt             1.5-6      2020-02-03 [1] CRAN (R 3.6.2)                   
#>  ModelMetrics       1.2.2.2    2020-03-17 [1] CRAN (R 3.6.2)                   
#>  modelr             0.1.6      2020-02-22 [1] CRAN (R 3.6.2)                   
#>  mRMRe              2.1.0      2020-01-10 [1] CRAN (R 3.6.1)                   
#>  munsell            0.5.0      2018-06-12 [1] CRAN (R 3.6.0)                   
#>  mvtnorm          * 1.1-0      2020-02-24 [1] CRAN (R 3.6.2)                   
#>  nlme               3.1-145    2020-03-04 [4] CRAN (R 3.6.2)                   
#>  nnet             * 7.3-13     2020-02-25 [4] CRAN (R 3.6.2)                   
#>  nnls               1.4        2012-03-19 [1] CRAN (R 3.6.0)                   
#>  np                 0.60-10    2020-02-06 [1] CRAN (R 3.6.2)                   
#>  PairedData       * 1.1.1      2018-06-02 [1] CRAN (R 3.6.2)                   
#>  pamr               1.56.1     2019-04-22 [1] CRAN (R 3.6.1)                   
#>  partykit           1.2-7      2020-03-06 [1] CRAN (R 3.6.2)                   
#>  pbivnorm           0.6.0      2015-01-23 [1] CRAN (R 3.6.1)                   
#>  pillar             1.4.3      2019-12-20 [1] CRAN (R 3.6.1)                   
#>  pkgbuild           1.0.6      2019-10-09 [1] CRAN (R 3.6.1)                   
#>  pkgconfig          2.0.3      2019-09-22 [1] CRAN (R 3.6.1)                   
#>  pkgload            1.0.2      2018-10-29 [1] CRAN (R 3.6.0)                   
#>  plotly           * 4.9.2      2020-02-12 [1] CRAN (R 3.6.2)                   
#>  plyr             * 1.8.6      2020-03-03 [1] CRAN (R 3.6.2)                   
#>  prettyunits        1.1.1      2020-01-24 [1] CRAN (R 3.6.2)                   
#>  pROC             * 1.16.1     2020-01-14 [1] CRAN (R 3.6.2)                   
#>  processx           3.4.2      2020-02-09 [1] CRAN (R 3.6.2)                   
#>  prodlim            2019.11.13 2019-11-17 [1] CRAN (R 3.6.1)                   
#>  profileR         * 0.3-5      2018-04-19 [1] CRAN (R 3.6.2)                   
#>  promises           1.1.0      2019-10-04 [1] CRAN (R 3.6.1)                   
#>  ps                 1.3.2      2020-02-13 [1] CRAN (R 3.6.2)                   
#>  psych              1.9.12.31  2020-01-08 [1] CRAN (R 3.6.2)                   
#>  purrr            * 0.3.3      2019-10-18 [1] CRAN (R 3.6.1)                   
#>  quadprog           1.5-8      2019-11-20 [1] CRAN (R 3.6.1)                   
#>  quantmod           0.4-16     2020-03-08 [1] CRAN (R 3.6.2)                   
#>  quantreg           5.54       2019-12-13 [1] CRAN (R 3.6.1)                   
#>  R.methodsS3        1.8.0      2020-02-14 [1] CRAN (R 3.6.2)                   
#>  R.oo               1.23.0     2019-11-03 [1] CRAN (R 3.6.1)                   
#>  R.utils            2.9.2      2019-12-08 [1] CRAN (R 3.6.2)                   
#>  R6                 2.4.1      2019-11-12 [1] CRAN (R 3.6.1)                   
#>  randomForest       4.6-14     2018-03-25 [1] CRAN (R 3.6.2)                   
#>  RColorBrewer     * 1.1-2      2014-12-07 [1] CRAN (R 3.6.0)                   
#>  Rcpp             * 1.0.4      2020-03-17 [1] CRAN (R 3.6.2)                   
#>  readr            * 1.3.1      2018-12-21 [1] CRAN (R 3.6.1)                   
#>  readxl             1.3.1      2019-03-13 [1] CRAN (R 3.6.0)                   
#>  recipes            0.1.9      2020-01-07 [1] CRAN (R 3.6.1)                   
#>  remotes            2.1.1      2020-02-15 [1] CRAN (R 3.6.2)                   
#>  reprex             0.3.0      2019-05-16 [1] CRAN (R 3.6.1)                   
#>  reshape          * 0.8.8      2018-10-23 [1] CRAN (R 3.6.0)                   
#>  reshape2           1.4.3      2017-12-11 [1] CRAN (R 3.6.0)                   
#>  rgl                0.100.50   2020-02-21 [1] CRAN (R 3.6.2)                   
#>  rJava              0.9-11     2019-03-29 [1] CRAN (R 3.6.0)                   
#>  rlang              0.4.5      2020-03-01 [1] CRAN (R 3.6.2)                   
#>  rmarkdown        * 2.1        2020-01-20 [1] CRAN (R 3.6.2)                   
#>  ROCR               1.0-7      2015-03-26 [1] CRAN (R 3.6.1)                   
#>  ROSE             * 0.0-3      2014-07-15 [1] CRAN (R 3.6.2)                   
#>  rpart              4.1-15     2019-04-12 [1] CRAN (R 3.6.2)                   
#>  rprojroot          1.3-2      2018-01-03 [1] CRAN (R 3.6.0)                   
#>  rsq              * 1.1        2018-09-27 [1] CRAN (R 3.6.2)                   
#>  rstudioapi         0.11       2020-02-07 [1] CRAN (R 3.6.2)                   
#>  rvest              0.3.5      2019-11-08 [1] CRAN (R 3.6.1)                   
#>  RWeka              0.4-42     2020-02-02 [1] CRAN (R 3.6.2)                   
#>  RWekajars          3.9.3-2    2019-10-19 [1] CRAN (R 3.6.1)                   
#>  scales           * 1.1.0      2019-11-18 [1] CRAN (R 3.6.1)                   
#>  sessioninfo        1.1.1      2018-11-05 [1] CRAN (R 3.6.0)                   
#>  shiny              1.4.0.2    2020-03-13 [1] CRAN (R 3.6.2)                   
#>  SparseM            1.78       2019-12-13 [1] CRAN (R 3.6.1)                   
#>  stabs              0.6-3      2017-07-19 [1] CRAN (R 3.6.0)                   
#>  stringi            1.4.6      2020-02-17 [1] CRAN (R 3.6.2)                   
#>  stringr          * 1.4.0      2019-02-10 [1] CRAN (R 3.6.2)                   
#>  survival         * 3.1-11     2020-03-07 [1] CRAN (R 3.6.2)                   
#>  testthat           2.3.2      2020-03-02 [1] CRAN (R 3.6.2)                   
#>  tibble           * 2.1.3      2019-06-06 [1] CRAN (R 3.6.2)                   
#>  tidyr            * 1.0.2      2020-01-24 [1] CRAN (R 3.6.2)                   
#>  tidyselect         1.0.0      2020-01-27 [1] CRAN (R 3.6.2)                   
#>  tidyverse        * 1.3.0      2019-11-21 [1] CRAN (R 3.6.2)                   
#>  timeDate           3043.102   2018-02-21 [1] CRAN (R 3.6.0)                   
#>  TTR                0.23-6     2019-12-15 [1] CRAN (R 3.6.1)                   
#>  usethis          * 1.5.1      2019-07-04 [1] CRAN (R 3.6.0)                   
#>  vctrs              0.2.4      2020-03-10 [1] CRAN (R 3.6.2)                   
#>  viridisLite        0.3.0      2018-02-01 [1] CRAN (R 3.6.0)                   
#>  webshot            0.5.2      2019-11-22 [1] CRAN (R 3.6.1)                   
#>  withr              2.1.2      2018-03-15 [1] CRAN (R 3.6.0)                   
#>  xfun               0.12       2020-01-13 [1] CRAN (R 3.6.1)                   
#>  xml2               1.2.5      2020-03-11 [1] CRAN (R 3.6.2)                   
#>  xtable             1.8-4      2019-04-21 [1] CRAN (R 3.6.0)                   
#>  xts                0.12-0     2020-01-19 [1] CRAN (R 3.6.2)                   
#>  yaml               2.2.1      2020-02-01 [1] CRAN (R 3.6.2)                   
#>  zoo                1.8-7      2020-01-10 [1] CRAN (R 3.6.1)                   
#> 
#> [1] /home/konrad/R/x86_64-pc-linux-gnu-suppressMessages(library/3.6)
#> [2] /usr/local/lib/R/site-suppressMessages(library)
#> [3] /usr/lib/R/site-suppressMessages(library)
#> [4] /usr/lib/R/suppressMessages(library
)

To render this tutorial we used:

knit("Tutorial.Rmd", "Tutorial.html")